gadreel/divconq

File Transfer Server Framework

divconq's Issues

Squad vs Team

Clean up terminology - Squad vs Team

SysAdmin Tools: SQL

Need a web UI for full access to DB. Consider also integrating something like this:

http://mistic100.github.io/jQuery-QueryBuilder/

OSv Integration

Add to the OSv GUI for monitoring, also review the tracing features:

http://osv.io/blog/blog/2014/09/18/cassandra-dashboard-tab/

Add CSP to dcWeb

content security policy - strict by default

Interesting case study

http://java.dzone.com/articles/io-files-arent-files

Hopefully we can do something like this someday

Solo Workqueue

We have a SQL implementation for workqueue, but sometimes we run without a SQL backend. There should be a solo workqueue implementation that defaults in if no other is used. Does not persist or share work, but does retry and other workqueue like things.

dcFileServer Intake Processing

1. Client - request Intake Id for given Intake Path
2. Client - upload one or more files with Intake Id, provide evidence for each
3. Client - after all files are contributed and accepted, indicate that the Intake is ready
4. Server - run trackable (client can see progress) processing on the Intake

Intake processing is always trackable because, by definition, the upload is not complete until intake processing is complete. If processing occurs after Intake that is a different story. So deposit validation may occur during Intake or later, but if the former the client has the option of reviewing the progress in near real time.

Individual files may also have Intake paths - processing that occurs on the file before the upload is complete. If the individual file is part of an Intake then the file's intake is processed first, then the collective intake.

TAR like HTTP Upload Stream (Common Transfer Protocol)

Can we adopt an upload channel so that it accepts a TAR like stream (single HTTP upload connection) that untars as it is delivered?

An on the fly TAR + GZ stream could deliver a large number of small files quickly.

Build DivConq

add Gradle build scripts and provide instructions

docker demo

Create and publish demo in Docker with Zulu:

http://betterjava.wordpress.com/2014/09/23/zulu-our-build-of-openjdk-is-live-on-docker/

Add and support slf4j

Some dependency libraries need slf4j-api. We want those libraries to log to dcLogger, so add slf4j and then implement their logger in dcLogger.

dcScript Web Interface

SysAdmin can load and run or debug dcScripts via web interface

dcbus and network diagnositcs

create tools for testing connections, protocols, bus. see divconq.tool.bus.Diagnostic as a start.

support alternative ssl engines

http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#34.0

Consider Java 8 DateTime

does it do all that JodaTime does, like ISO 8601 periods?

High Performance Links

Someday we want DivConq to offer connections to or embed some sort of high performance distributed processing like Hadoop or HPPC

Stamped Locks

Tested StampedLock in divconq.work.Worker. Like it. Try it other places in DC too.

more info:
http://javaspecialists.eu/talks/jfokus13/PhaserAndStampedLock.pdf

Groovy Web Shell

Sessions with root privileges can accept remote groovy commands

CA trust management

Setup hardening so that:

• ignore signing chain, assume self signed - default, trust is done at bus and surface level
• trust specific thumbprints (certs) regardless of self signed or not
• trust the big CAs and their chains

Additional flag:

• trust CACert flag - off by default

netty performance part 2

Tried using same thread group for dcBus and for HTTP (surfaces) and it didn't work, occasionally got blocking exception writing to bus while handling data from HTTP even with just 1 upload going (typically during the first block of upload). this is on netty 4.0.23. try to improve this later...try:

• same thread group in higher version
• without sync in bus
• turn off auto read from HTTP and read only as needed (as available)

netty performance fixes

Fix some ways we use Netty, especially use pooled buffers:

http://normanmaurer.me/presentations/2014-facebook-eng-netty/slides.html#14.0

Initial FileOps

[FileOps](/Gadreel/divconq/wiki/Feature FileOps) feature in [dcScript](/Gadreel/divconq/wiki/Feature dcScript)

Support File Instructions

• FileStore
• File
• Folder
• LocalFile
• LocalFolder
• TempFile
• TempFolder

Support Ops Commands

• Tar
• Untar
• Gzip
• Ungzip
• Split
• Join
• Copy
• XCopy
• Delete
• PGPEncrypt

FileStore interface cleanup

Much of the IFileStore* interface is dead code, along with FileSys* - cleanup and make it functional

html editor/cli/syntax

http://ace.c9.io/#nav=about

https://github.com/joewalker/gcli

https://github.com/chjj/marked

Deposit Tracking Module

We need a deposit tracking module:

1. When is deposit due?
2. What to do when deposit missed?
3. Who to inform when deposit is missed?
4. How to identify a complete deposit (use Contributions, one or more contribution can be made to complete a full deposit)
5. Where are contributions uploaded?
6. How does the contribution trigger review of "completeness"
7. Where are contributions stored? May be moved from upload. May also be transformed.
8. How to trigger validation of deposit, if required?
9. Who to inform if validation is valid, invalid, incomplete?

Support more FileOps

[FileOps](/Gadreel/divconq/wiki/Feature FileOps) feature in [dcScript](/Gadreel/divconq/wiki/Feature dcScript)

Support File Instructions

• ZipFileStore
• SftpFileStore
• Select

Support Ops Commands

• Wget
• Touch
• PGPDecrypt
• PGPSign
• PGPVerify

SFTP server support

Add support for SFTP interface to a FileStore, via Gateway when used.

default SSL protocol document - harden

By default we are setting support to TLS 1.2 only, because most browsers that support TLS 1.1 also support 1.2. This excludes a lot of older browsers though:

http://en.wikipedia.org/wiki/Transport_Layer_Security#Web_browsers

Proto: SSLv2Hello
Proto: SSLv3
Proto: TLSv1
Proto: TLSv1.1
Proto: TLSv1.2

Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
Suite: TLS_RSA_WITH_AES_256_CBC_SHA256
Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
Suite: TLS_RSA_WITH_AES_256_CBC_SHA
Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA
Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA
Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
Suite: TLS_RSA_WITH_AES_128_CBC_SHA256
Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
Suite: TLS_RSA_WITH_AES_128_CBC_SHA
Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA
Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA
Suite: TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
Suite: TLS_ECDHE_RSA_WITH_RC4_128_SHA
Suite: SSL_RSA_WITH_RC4_128_SHA
Suite: TLS_ECDH_ECDSA_WITH_RC4_128_SHA
Suite: TLS_ECDH_RSA_WITH_RC4_128_SHA
Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
Suite: TLS_RSA_WITH_AES_256_GCM_SHA384
Suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
Suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
Suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Suite: TLS_RSA_WITH_AES_128_GCM_SHA256
Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
Suite: SSL_RSA_WITH_3DES_EDE_CBC_SHA
Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
Suite: SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
Suite: SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
Suite: SSL_RSA_WITH_RC4_128_MD5
Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV

Protect UserContext Password

Password from UserContext should not appear in logger

dcScript internal vars

Get _LastResult working such that a call to a function sets _LastResult for the calling code.

Var, With and Global should all set _LastResult to the target.

FileOps should set _LastResult to the destination stream.

CtpSession/CtpSend should set _LastResult

Shell and Email should set _LastResult.

probably make _LastResult, _LastCode, _Errored, _ExitCode, _Now and _Log global vars

Session Tether

FileStore services, for one example, need to also have access to a Session for caching. If the client connects to the dcFileServer this is easy, the FileStore service just looks up the local session. But if the client connects to dcFileGateway then the user's session is on the Gateway. We need a way for the FileStore service to tether to the gateway session so that it keeps alive as long as the gateway session does. The tether session will be managed same as other sessions on dcFileServer, so should more than one service require access to the session there is only one instance of the tethered session per backend server.

Gateway's session should keep alive as long as the backends session is active and vice versa.

stream debugging

create a funnel stream that allows only N number of bytes to pass at a time, then place it before each other stream and see if the decoders blow up. also use with netty direct stream

WebDAV Access for root domain

Be able to access the web files of the server for quick remote edits

Encryption and Compression Abstraction

Provide a pluggable system for file encryption methods and file compression methods - interfaced with FileStore abstraction.

Minimally be able to encrypt/decrypt/sign with GPG via command line or PGP implemented in Java.

Minimally be able to compress/decompress with 7zip via command line or Apache Compression implemented in Java.

Part One

Provide [FileOps](/Gadreel/divconq/wiki/Feature FileOps) feature in dcScript. See issue #20.

Support epoll in Linux

https://github.com/netty/netty/wiki/Native-transports

dcScript support for MS documents

based on http://poi.apache.org/ if possible

Locale support in CommonPath, dcScript, etc

currently path names cannot contain none ASCII (7 bit) characters, and even then only limited. [make sure we cannot contain control characters, ftw). include in dcScript is very limited too.

also dcScripts formatting is not based on OperationContext, give dcSdcript more OC smarts.

Maven support

pom file

jars in maven central

Web UI for Demo File Server

FileStore navigation, upload and download support

Timestamping service for signing

Develop a dc timestamping service to provide an independent party with a reliable time stamp service for use in signing files.

The timestamping service stores the signature (enough to verify the signature in question, at least the hash) and the timestamp the service provided. This makes it possible for another party check that the signature was indeed performed at the time the submitted signature claims.

Allow other time stamping services to be used. Review time stamping service in this article that mentions PGP Digital Timestamping Service

syslog integration

add support so we can call syslog

In FileOps support AWK like functionality

Good tutorial:

http://www.grymoire.com/Unix/Awk.html

Process text as columns (fields) from each line (record).

Default field separator is 2 or more spaces - default records separator is \n. Each field has leading and trailing space trimmed. Field and Record separator can be overriden.

Services Navigator

Web UI for navigating the available services.

Add support for CIFS

CIFS/SMB support will be useful when running on OSv or other system where configuring CIFS natively is challenging or impossible.

http://jcifs.samba.org/

review:

http://sourceforge.net/projects/webdisk/
http://sourceforge.net/p/webdisk/wiki/Home/

use dot temp approach

Include a configuration option in FileStore to use a dot something approach with files in upload, renaming after done with upload (on successful evidence). This could be:

<Settings FileStorePath="D:\temp\simplefs" BestEvidence="SHA512" 
  UploadExtension=".partial"
/>

Or such so that sweepers can identify files that are incomplete and ignore them,

Add RegEx pattern logic to StringStruct

so can use in If and such

stream support for direct buffers

currently direct buffers might cause issues with file streams...use this approach (in UngzipStream) to support others...

        int readableBytes = in.readableBytes();

        if (in.hasArray()) {
            this.inflater.setInput(in.array(), in.arrayOffset() + in.readerIndex(), readableBytes);
        } 
        else {
            byte[] array = new byte[readableBytes];
            in.getBytes(in.readerIndex(), array);
            this.inflater.setInput(array);
        }

Multi-factor Authentication

Build in support for things like:

http://www.smspasscode.com/

http://www.deepnetsecurity.com/authenticators/

or even just plain SMS with our own codes:

http://mblox.com/products/sms

communication features

chat, video conference, present - a la Big Blue Button features...

plus be able to share files real time or to In Box - with secure messages.

See Smart Box for examples.

RocksDB Integration

Use the Java libs for RocksDB to provide an alternative database (small data storage) to SQL. Use the design I had for MUMPS - Rocks will be the "globals" and Groovy will be the stored procedures.