KafkaLogOnEvent handler fails to find configured LogFormat

[shunshunbu]

KafkaLogOnEvent handler fails to find configured LogFormat

[shunshunbu]

KafkaEngine on KafkaLog /var/log/ftpd/kafka.log KafkaBroker 172.0.0.1:9092
LogFormat kafka "%h %l %u %t "%r" %s %b" KafkaLogOnEvent ALL kafka mytopic

when i start proftpd,there is an error: fatal : kafkaLogOnEvent : no LogFormat 'kafka' configured on line 37 of '/usr/local/proftpd/etc/proftpd.conf' ,is there anything wrong?

the proftpd version is 1.3.8b
use this configure: ./configure --prefix=/usr/local/proftpd --enable-nls --enable-ctrls --with-modules=mod_kafka:mod_proxy:mod_ctrls_admin:mod_ifsession

[Castaglia]

Can you provide the full ProFTPD configuration you using, please, not just excerpts?

[shunshunbu]

the proftpd.conf

this is full conf:

`
# This sample configuration file illustrates creating two
# virtual servers, and associated anonymous logins.

ServerName			"ProFTPD"
ServerType			standalone

# Port 21 is the standard FTP port.
Port				21

# Don't use IPv6 support by default.
UseIPv6             off

# Global creates a "global" configuration that is shared by the
# main server and all virtualhosts.

<Global>
  # Umask 022 is a good standard umask to prevent new dirs and files
  # from being group and world writable.
  Umask				022    
  SysLogLevel       DEBUG
  SystemLog         /var/log/proftpd/system.log
  ExtendedLog       /var/log/proftpd/extended.log ALL
  TransferLog       /var/log/proftpd/xfer.log
  RequireValidShell off


  <IfModule mod_kafka.c>
    KafkaEngine  on
    KafkaLog     /var/log/proftpd/kafka.log
    KafkaBroker  127.0.0.1:9092
    LogFormat    kafka "%h %l %u %t %r %s %b"
    KafkaLogOnEvent ALL kafka topic bss 
  </IfModule>
</Global>

ProxyTables       /var/ftp/proxy
TraceLog          /var/log/proftpd/proxy-trace.log

Trace proxy:100 proxy.ftp.data:100 data:100 fsio:100 command:100 event:100 proxy.ftp.ctrl:100 proxy.ftp.conn:100 proxy.ftp.xfer:100 proxy.ftp.msg:100 bss:100 proxy.reverse:100 netacl:100 kafka:100

<Class virtualhost1>
  From 10.20.17.203
  #From 10.20.25.189
</Class>



# Set the user and group that the server normally runs at.
User				root
Group				root

# To prevent DoS attacks, set the maximum number of child processes
# to 30.  If you need to allow more than 30 concurrent connections
# at once, simply increase this value.  Note that this ONLY works
# in standalone mode, in inetd mode you should use an inetd server
# that allows you to limit maximum number of processes per service
# (such as xinetd)
MaxInstances                    30

# Maximum seconds a data connection may "stall"
TimeoutStalled			300



<VirtualHost 10.20.77.23>
    ServerName           "Virtual.com's FTP Server"
    Port                 7000
    MaxClients           10
    MaxLoginAttempts     1

   <Limit LOGIN>
     Order deny,allow
     Allow from 10.20.25.189
   </Limit>
    
    <IfUser proftpdtest>     
        <Limit LOGIN>
            DenyAll 
        </Limit>
    </IfUser>

    <Limit LIST CDUP CWD PWD XCWD XCUP >
        
    </Limit>

    <IfModule mod_proxy.c>
        ProxyEngine on
        ProxyLog /var/log/proftpd/proxy.log
        ProxyRole reverse
        ProxyReverseConnectPolicy RoundRobin
        ProxyDataTransferPolicy client
        ProxyReverseServers ftp://10.20.25.210:21
    </IfModule>

</VirtualHost>

# First virtual server
<VirtualHost 10.20.77.23>
  ServerName			"Virtual.com's FTP Server"
  Port                  6000
  MaxClients			10
  MaxLoginAttempts		1

  # DeferWelcome prevents proftpd from displaying the servername
  # until a client has authenticated.
  DeferWelcome			on


  # Limit normal user logins.
  #<Limit LOGIN>
  #  Order deny,allow
  #  AllowUser   cust1
  #  #AllowGroup  custgroup
  #  DenyAll
  #</Limit>

  <Directory /zfdata/bss/test/proftpdi>

  </Directory>

  <Limit ALL>
    AllowClass virtualhost1
    DenyAll
  </Limit>

  <Limit LOGIN>
    DenyUser bob
  </Limit>

  <IfUser cust1>

    <Limit LOGIN>
      #Order deny,allow
      AllowClass virtualhost1
      DenyAll
    </Limit>

    <Limit ALL>
      AllowAll
    </Limit>

    <Limit LIST CDUP CWD PWD XCWD XCUP>
      AllowAll
    </Limit>

    <Limit STOR STOU>
      AllowAll
    </Limit>

    <Limit RETR>
      DenyAll
    </Limit>
 
   MaxStoreFileSize 1 b user cust1

  </IfUser>


 <IfModule mod_proxy.c>
  ProxyEngine on
  ProxyLog /var/log/proftpd/proxy.log
  ProxyRole reverse
  ProxyReverseConnectPolicy RoundRobin
  ProxyDataTransferPolicy client
  #ProxyReverseServers ftp://10.20.77.26:21
  ProxyReverseServers ftp://10.20.25.210:21
 </IfModule>
  # Next, create a "guest" account (which could be used
  # by a customer to allow private access to their web site, etc)
  <Anonymous ~cust2>
    User			cust2
    Group			cust2
    AnonRequirePassword		on

    <Limit LOGIN>
      AllowAll
    </Limit>

    HideUser			root
    HideGroup			root

    # A private directory that we don't want the user getting in to.
    <Directory logs>
      <Limit READ WRITE DIRS>
        DenyAll
      </Limit>
    </Directory>
  </Anonymous>
</VirtualHost>

# Another virtual server, this one running on our primary address,
# but on port 4000.  The only access is to a single anonymous login.
<VirtualHost 10.20.77.23>
  ServerName			"Our private FTP server"
  Port				5000
  Umask				027

 <IfModule mod_proxy.c>
  ProxyEngine on
  ProxyLog /var/log/proxy.log
  ProxyRole reverse
  ProxyReverseConnectPolicy RoundRobin
  ProxyDataTransferPolicy client
  ProxyReverseServers ftp://10.20.77.26:21
 </IfModule>

  <Limit LOGIN>
    #DenyAll
    AllowAll
  </Limit>

  <Anonymous /usr/local/ftp/virtual/a_customer>
    User			ftp
    Group			ftp
    UserAlias			anonymous ftp

    <Limit LOGIN>
      AllowAll
    </Limit>

    <Limit WRITE>
      DenyAll
    </Limit>

    <Directory incoming>
      <Limit WRITE>
        AllowAll
      </Limit>
    </Directory>
  </Anonymous>
</VirtualHost>


#proxyId_begin:10
<VirtualHost 10.20.77.23>
  ServerName        Virtual.com's FTP Server
  Port              9000
  MaxClients        10
  MaxLoginAttempts  1

  <Limit LOGIN>
      Order deny,allow
      Allow from 10.20.25.189
      Allow from 192.168.0.0/16
  </Limit>

  <IfUser bob>
      <Limit LOGIN>
          DenyAll
      </Limit>
  </IfUser>

  <IfUser cust1>
      <Limit LOGIN>
          DenyAll
      </Limit>
  </IfUser>

  <Limit LIST OPEN DELETE >
      DenyAll
  </Limit>

  <IfModule mod_proxy.c>
      ProxyEngine                 on
      ProxyLog                    /var/log/proftpd/proxy.log
      ProxyRole                   reverse
      ProxyReverseConnectPolicy   RoundRobin
      ProxyDataTransferPolicy     client
      ProxyReverseServers         ftp://10.20.25.210:21
  </IfModule>

</VirtualHost>

#proxyId_end:10



#proxyId_begin:10
<VirtualHost 10.20.77.23>
  ServerName        Virtual.com's FTP Server
  Port              9876
  MaxClients        10
  MaxLoginAttempts  1

  <Limit LOGIN>
      #Order allow,deny
      Allow from 10.20.25.189
      DenyUser cust1
  </Limit>



  <IfModule mod_proxy.c>
      ProxyEngine                 on
      ProxyLog                    /var/log/proftpd/proxy.log
      ProxyRole                   reverse
      ProxyReverseConnectPolicy   RoundRobin
      ProxyDataTransferPolicy     client
      ProxyReverseServers         ftp://10.20.25.210:21
  </IfModule>

</VirtualHost>

#proxyId_end:10
`

[Castaglia]

I see the issue here. The mod_kafka module's KafkaLogOnEvent handler tries to ensure that the configured LogFormat name exists, at startup/parse time. But when a <Global> configuration section is used, the handler can't easily find that LogFormat configuration.

Use of a <Global> section should be properly supported; this is a bug in the mod_kafka module. I'll see what I can do to handle this configuration better.

[Castaglia]

This should now be fixed in master. Thanks!