channelaccess/ca

Java Channel Access Client Implementation

ca's Issues

Write a monitor notification service impl that buffers to ensure that notifications are not dropped (eg suitable for archiver)

Provide the ability for library users to configure the appropriate monitor notification strategies

This tracker is to migrate the CA library implementation towards a more flexible, more configurable Monitor Notification solution that satisfies our current needs at PSI and hopefully those of our external users.

Let's first establish the requirements that any MonitorNotificationServiceImpl must satisfy. These were first stated in a comment on this earlier tracker. Following feedback from Simon and Fabian (thanks guys :-) ) I would now restate them as follows:

Monitor Notification Service - Requirements

1. A slow Consumer should not stop other Consumers from eating
   What does this mean ? A fast update rate on one monitored channel should not result in other Consumers getting starved of data. Obviously, the CPU is not an infinite resource so when excessive demands are made on the notification system it has only limited choices. The more concrete requirements should be taken as follows:
   • (a) a single slow Consumer should not block all other Consumers from being notified.
   • (b) CPU bandwidth should be shared fairly by default.
   • (c) CPU bandwidth should be shared appropriately when the library user has provided some additional usage hints.
2. The system should cater for both Lossy and Non-Lossy Consumers. See Additional Notes.
3. If a Lossy Consumer is too slow to eat then it is acceptable to silently drop intermediate notification values. However:
   • (a) values must always be sent in the order in which they were received from the TCP/IP monitor update stream, and
   • (b) after a stream of notifications the Consumer must always finally be sent the last recently updated value.
4. If a Non-Lossy Consumer is too slow to eat then data must be buffered until it can be delivered. Clearly there are limits to this, so the following additional rules apply:
   • (a) the buffer should be of unlimited size by default.
   • (b) the buffer can optionally be of limited size if the library user has provided some sizing hints.
5. When an attempt is made to publish information to a Non-Lossy Consumer with a full buffer the default policy will be to drop the oldest value. This should not be done silently - the library return code should make it clear that information was lost.
6. Consumers should be sent values as closely as possible in the sequence that the values were received from the TCP/IP monitor update stream. This keeps latency as low as possible.
7. It should be possible to monitor at least 100_000 channels without running into performance bottlenecks. Since most JVM installations have a resource limit of only a few thousand threads this means that it is impractical for implementations to dedicate a single thread per monitor.
8. Monitor notification comes from EPICS IOCs. With todays scan rates it is highly unlikely that the event rate from any IOC will exceed 1 event/ms. However, the CA client library should potentially be able to cope with data streaming from 100,000 PV's. This puts an estimate on the upper bound of required performance of 100,000 * 1000 * 24 bytes/second = 24GB/second. This exceeds the network bandwidth of Gigabit ethernet so for all practical purposes the processing performance requirement is determined by the network bandwidth itself.
9. The data associated with each IOC event may vary typically from the CA minimum header size (16 bytes) through the size associated with the transfer of simple data types (8 extra bytes) to the size associated with a camera image (several megabytes). It is believed that the transfer of camera data can get close to, or even exceed the Gigabit network bandwidth. This is probably somewhere around 120 MBytes/second. See here for a discussion on this.

Additional Notes:

1. Lossy Consumer example: a GUI application in the WBGB Control Room.
2. Non-Lossy Consumer example: a Data Archiver application eg for SwissFEL.

Introduction of default timeouts for synchronous functions

For synchronous functions like get(), put(), connect() there should probably a default timeout instead of blocking forever.
This timeout should be configureable via context properties, I guess ...

Bulk creation of channels

Besides the current base methods to create channels I guess there should be some convenience method to bulk create channels. This method should accept a list/set/map of channels and create and connect all of these.

We could somehow adopt the way jcae does it right now, i.e. via ChannelDescriptor (this current solution however needs to be revised somehow though).
If we go that route (channel descriptor), eventually we change the base methods as well to accept a channel descriptor ... - however then we have to distinguish the methods that automatically connect the channels and which do not ...

FIx closedown bug

Under "certain circumstances" (Simon Ebner encountered it with the PSI snapshot application) the CA library emits "nasty messages" to the console. (I deliberately state this ticket in vague terms because currently the ticket is just a placeholder which requires more substantiation...)

This tracker is to investigate and where necessary make a fix.

Specifying Unicast Address in ADDR_LIST does not work

When specifying a unicast address in the address list properties the channel will not be found.(It works if specifying the broadcast address)

This is the test code I used (the channel ARIDI-PCT:CURRENT is on the machine 129.129.130.20 ):

public static void main(String[] args) throws InterruptedException, ExecutionException, TimeoutException {
        Properties properties = new Properties();
        properties.setProperty(Constants.ADDR_LIST_KEY, "129.129.130.20");
//      properties.setProperty(Constants.ADDR_LIST_KEY, "129.129.130.255"); // this works
        try (Context context = new Context(properties))
        {
          Channel<Double> channel = context.createChannel("ARIDI-PCT:CURRENT", Double.class);
          channel.connectAsync().get(1, TimeUnit.SECONDS);
          System.out.println(channel.getProperties());
          System.out.println(channel.get());
          channel.close();
        }
    }

Convenience Functions - waitFor()

while working with the ca library I got the feeling it might be good to add some more convenience functions the Channel interface.
The methods I think of are:

waitFor(T value);
waitFor(T value, Comparator comparator);

CompletableFuture f = waitForAsync(T value);
CompletableFuture f = waitForAsync(T value, Comparator comparator);

Regarding the names of the function I am not sure whether to just use waitFor or waitForValue .

“Within" these functions what I would do is, attaching a monitor, waiting on the value and then detaching the monitor ...

What do you think of this?
Would this make sense?

Provide direct access to socket ByteBuffer through MonitorNotifcationService interface

Opens the path to more efficient implementations

Library does not interoperate with CA in EPICS 7.0 - "client version too old"

This was reported by one of PSI's beamline scientists when working with the MATLAB wrapper to CA.

Here is the fault report:

The problem seems that the Matlab channel access client has problem to communicate with an EPICS7 IOC on sf-ecmc-04.psi.ch

Sometimes Franziskas script just works but sometimes it just hangs when opening the connection to the IOC

What can be seen on the IOC and what we see on the client side is that there is a message that the client version is too old.

On the IOC shell:

sf-ecmc-04> CAS: request from 129.129.145.206:54086 => CAS: Client version too old
CAS: Request from 129.129.145.206:54086 => cmmd=23 cid=0x0 type=0 count=0 postsize=0
CAS: Request from 129.129.145.206:54086 => available=0x0 N=0 paddr=0x7f66c800d968

On the client side - if the connection is established and the caget works:

Jan 30, 2020 10:17:05 AM org.epics.ca.impl.ResponseHandlers exceptionResponse
WARNING: Exception message reported, code: DEFUNCT, message: 'CAS: Client version 0 too old'.

Where can I find CA api?

Add support for StripedExecutorService

Create a MonitorNotificationServiceImpl with shared disruptor

The goal would be to see whether we can come up with a Disruptor solution that is both more performant than a standard Java Blocking Queue based approach and yet which doesn't use one thread per consumer. The Disruptor Polling feature probably provides the key to this, although documentation is rather ppor on this feature.

Deploy to Maven Central

Decide on whether to keep possibility to send special monitor values (currently null) when a channel disconnects

Convenience Class - Monitored Channel

Class/Object that holds cached value that gets updated by a monitor ...
(create this object e.g. via the Channels utility class ...)

Some Units like micro does not show up correctly

Some special units of channels are not retrieved correctly.
For example micro (µ) ends up in char 65533 (�)

Upgrade to Junit5

Merge AlarmSeverity and AlarmStatus Enumerations into Alarm Class

Merge the external AlarmSeverity and AlarmStatus enumerations into the Alarm class so that they can be referred to as: Alarm.AlarmStatus
I guess this is more clean and simpler

Transition away from 'travis-ci.org' site to newer 'travis-ci.com' site.

Use of the .org site is now discouraged so transitioning to the new .com one.

Fix monitor scalability bottleneck

The current implementation of the CA channel monitoring feature has a "scalability bottleneck" in that every monitor creates a separate thread to deal with notification. This imposes an upper limit on the number of monitors that can be created in the client library. On my system (MacBook Pro) the upper limit is around 4000.

This task is to investigate the issue and come up with a modified implementation where this limitation does not exist.

Create CA Software Release 1.2.3

Readme unclear about supported listeners

The file Readme.md contains the following item in the Features section:

Support of all listeners ChannelAccess supports: ConnectionListener, AccessRightListener, Value Listener (Monitor)

It's unclear to me what that means. Is it saying that it supports similar listening to what can be done in a JCA implementation?

JCA contains the interfaces ConnectionListener, AccessRightsListener, and MonitorListener (among others). If that's what this is referring to, then perhaps the item should read as follows:

Supports listening to all Channel Access events that can be listened to with a JCA implementation (e.g., similar to JCA's ConnectionListener, AccessRightListener, MonitorListener, etc.)

Upgrade for compatibility with latest versions of Java, Gradle, Disruptor, Junit etc

Add support for uploading builds to PSI artifactory

Monitor Maximum Update Rate

Add an option to the addMonitor function of a channel to specify the maximum update rate of a monitor.

Improve robustness of some tests.

Remove TAB character from source files

The use of TAB character mixed with spaces makes it difficult/impossible to view the source code in a consistent way on multiple platforms.

This task is to do a one shot change on all source files to eliminate TAB and to replace it with three spaces.

Monitored Channels

Have a flag to tell that a channel should cache values on the client side and if get() is called only the cached value get returned to the calling code.

... internally the channel would just automatically setup a value monitor and cache the returned values in a variable...

This functionality we could provide either within the current channel implementation or with a wrapper class (e.g MonitoredChannel(channel)) which has the same signature than a channel ...

Adapt project structure so that 'src/test/java' tree is congruent with 'src/main/java' tree

This provides the possibility to reduce the visibility of certain methods in the project.

Switched to openjdk8.

Create software release 1.2.2

Minor documentation improvements

Add statement on CA version compatibility. Add section for documents.

Getting Metadata for Generic (Object) Channel does not work

When trying to get more metadata to the value with an generic channel the library fails

Channel<Double> channel = context.createChannel("ARIDI-PCT:CURRENT", Double.class);
channel.connectAsync().get(1, TimeUnit.SECONDS);
System.out.println(channel.get(Timestamped.class));

The exception thrown is:

Exception in thread "main" java.lang.RuntimeException: Failed to do get.
    at org.epics.ca.impl.ChannelImpl.get(ChannelImpl.java:332)
    at org.epics.ca.test.TE.main(TE.java:24)
Caused by: java.lang.RuntimeException: unsupported channel metadata type class org.epics.ca.data.Timestamped<class java.lang.Object>
    at org.epics.ca.impl.ChannelImpl.getAsync(ChannelImpl.java:346)
    at org.epics.ca.impl.ChannelImpl.get(ChannelImpl.java:330)
    ... 1 more

Just getting the value works!

System.out.println(channel.get());

When doing the same with a typed channel things work:

Channel<Double> channel = context.createChannel("ARIDI-PCT:CURRENT", Double.class);
channel.connectAsync().get(1, TimeUnit.SECONDS);
System.out.println(channel.get(Timestamped.class));

Get the Example Program Working

The Example program does not currently (CA release 1.2.2) work and in many cases it is not clear to me the intention of the code.
This task is to at least get it going again.

Cleanup warnings detected by IntelliJ

Annotations

We should introduce the ability to annotate Channel declarations.
There should be a way to create/connect and close all of those channels within an object with annotations.

Macro substitution need to be implemented for this for sure.

Add initial support for deploying via bintray/jcenter...

Library fails to detect IOC connection state changes when operating through a firewall

Improvement nativeType property of a channel

While getting the property nativeType from a channel the return value is a number
It would be great if it would return the actual Java type:

channel.getProperties().get("nativeType")

6

ideally this would be
channel.getProperties().get("nativeType")

Double.class

or something like this ...

or is there a way to map the number to the type?

Revert to java.util.logging throughout

Update Readme to indicate relationship with CAJ

Should I be using Channel Access Java (CAJ) or this library. Is this library production ready? Is anyone using it? By the way the name "ca" is so generic it is tricky to search for. Maybe you should have given this library a clever name like CAJ2.

Investigate and where necessary fix bug suggested by Fabian with respect to out of order notifications

Create software release 1.2.0

Create software release 1.2.1

Create software release 1.1.0-RELEASE

Eliminate javadoc warnings

Get rid of all javadoc warnings .

Note: the goal here is to stop the stream of (over 100) ugly warning messages that get emitted when building the project. Writing good javadoc takes time and is hard. This issue does not seek to address that, but purely to improve things to the point where the build proceeds quietly. This will be accomplished by making small tweaks to the code. Unfortunately without a deep understanding of the underlying implementation it may sometimes be necessary to do things like this:

@param myThingy the thingy of mine.

One can argue that this is meaningless ! In time it is hoped that we can improve the Javadoc so that it provides something meaningful.

Fix bug in TCPTransport receive buffer processing

Whist working on PSI's wica project it was observed that caget bulk operations failed under certain circumstances. When the issue arose the following message was logged to the console:

Jun 01, 2018 12:35:40 AM org.epics.ca.impl.ResponseHandlers handleResponse
WARNING: Invalid response message (command = 13876) received from: /192.168.0.25:5064
Jun 01, 2018 12:35:40 AM org.epics.ca.impl.ResponseHandlers handleResponse
WARNING: Invalid response message (command = 12544) received from: /192.168.0.25:5064
Jun 01, 2018 12:35:40 AM org.epics.ca.impl.ResponseHandlers handleResponse

This tracker is to investigate and fix the problem.

Rename channel.connect() to channel.connectAsync()

To be consistent in naming channel.connect() should be renamed to channel.connectAsync() and there should be a new function named connect with does the blocking connect (i.e. channel.connectAsync().get())

Revert Java buildCompatibility target to Java 8

Create new release targeted for compatibility with Java 8 JVMs