krishnact/gaevfs

Automatically exported from code.google.com/p/gaevfs

gaevfs's Issues

Use Google Accounts API to provide directory security (GaeVfsServlet)

Provide a way within GaeVfsServlet to protect directories using the Google 
Accounts API:

  http://code.google.com/appengine/docs/java/users/overview.html

It should be possible to configure directories to be completely restricted 
(read/write) to authorized users, or to be read-only to non-authorized 
users.

Original issue reported on code.google.com by [email protected] on 5 Jul 2009 at 1:46

alternate API to mimic java.io

As an alternate to the Commons VFS API, a "wrapper" API that mimics java.io 
classes (File, FileOutputStream, FileWrite, RandomAccessFile, etc.) should be 
implemented. This will make it easier for programmer who are familiar with 
java.io to use GaeVFS.

Original issue reported on code.google.com by [email protected] on 14 Aug 2009 at 9:15

Invoking FileObject.getChildren() on root directory includes the root directory in the returned array

Invoking FileObject.getChildren() on the root directory includes the root 
directory in the returned array. This is probably happening because we're 
doing a query with the key of the root directory as the ancestor, and the 
root directory doesn't have a parent (so the ancestor-key query is 
matching the root directory key itself).

Original issue reported on code.google.com by [email protected] on 2 Jul 2009 at 1:47

revisit use of parent keys (entity groups) within block keys

The current implementation stores the block keys as a list of keys within a 
metadata property field. The advantage of this approach is that it's easy 
to do bulk gets of all of the block entities. The problems with this 
approach are:

 1) This creates an entity group for all of the block keys for a file, 
causing datastore puts to be serialized and limiting to 10 puts per entity 
group per second (what happens if a file has more than 10 blocks that need 
to be written all at once?)

 2) There's a limit of 500 elements in a property list (?) and there's also 
the 1MB limit on the size of the metadata entity; both of these will limit 
the number of block keys that can be stored within a single metadata 
entity.

Should we instead use a unique key for each block and not use the metadata 
key as the parent? This would remove the limitations of the current design, 
but require searching for blocks instead of get-by-key.

Original issue reported on code.google.com by [email protected] on 19 Oct 2009 at 7:30

needs atomic locks

http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api
/memcache/MemcacheService.html

"Except for the increment(java.lang.Object, long) method, this service does 
not offer atomicity guarantees. "

The code is using a 'put' and ADD_ONLY_IF_NOT_PRESENT to acquire the lock.  
Since put is not atomic you could have two hosts in the cloud could acquire 
the same lock.

Only the increment methods are atomic.



http://code.google.com/appengine/docs/java/javadoc/com/google/appengine/api
/memcache/MemcacheService.html#put(java.lang.Object, java.lang.Object, 
com.google.appengine.api.memcache.Expiration)

Original issue reported on code.google.com by [email protected] on 30 Apr 2010 at 9:32

use of protocol buffers instead of Java serialization

As a potential performance enhancement, investigate the use of protocol 
buffers in place of Java serialization for creating the write-behind task 
payload.

http://code.google.com/p/protobuf/

http://code.google.com/apis/protocolbuffers/docs/overview.html

http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking
(be sure to read comments)

http://whiteboardjunkie.wordpress.com/2009/09/14/serialization-options-
compared/

Original issue reported on code.google.com by [email protected] on 16 Sep 2009 at 10:44

java.lang.NoClassDefFoundError: java.net.URLStreamHandler is a restricted class

What steps will reproduce the problem?
1. configure app engine application to use h2 1.2.121 and latest gaevfs
from svn
2. start application
3. error stacktrace in log

What version of the product are you using? On what operating system?
svn version 356 of gae-vfs, h2 version 1.2.121

Please provide any additional information below.

Config:
web.xml

    <servlet>
        <servlet-name>gaevfs</servlet-name>

<servlet-class>com.newatlanta.appengine.servlet.GaeVfsServlet</servlet-class>
        <init-param>
            <param-name>dirListingAllowed</param-name>
            <param-value>true</param-value>
        </init-param>
        <init-param>
            <param-name>initDirs</param-name>
            <param-value>/gaevfs/data</param-value>
        </init-param>
        <init-param>
            <param-name>uploadRedirect</param-name>
            <param-value>/gaevfs/</param-value>
        </init-param>
    </servlet>
    <servlet-mapping>
        <servlet-name>gaevfs</servlet-name>
        <url-pattern>/gaevfs/*</url-pattern>
    </servlet-mapping>
    <servlet>
        <servlet-name>CachingDatastoreService</servlet-name>

<servlet-class>com.newatlanta.appengine.datastore.CachingDatastoreService</servl
et-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>CachingDatastoreService</servlet-name>
        <url-pattern>/_ah/queue/write-behind-task</url-pattern>
    </servlet-mapping>

    <listener>

<listener-class>com.newatlanta.appengine.servlet.GaeVfsServletEventListener</lis
tener-class>
    </listener>

    <listener>
        <display-name>h2 starter listener</display-name>
        <listener-class>org.h2.server.web.DbStarter</listener-class>
    </listener>
    <context-param>
        <param-name>db.url</param-name>

<param-value>jdbc:h2:gae://gaevfs/data/sessions;FILE_LOCK=NO;AUTO_SERVER=FALSE;D
B_CLOSE_ON_EXIT=FALSE</param-value>
    </context-param>
    <context-param>
        <param-name>db.user</param-name>
        <param-value>sa</param-value>
    </context-param>
    <context-param>
        <param-name>db.password</param-name>
        <param-value>sa</param-value>
    </context-param>


Stacktrace:

java.lang.NoClassDefFoundError: java.net.URLStreamHandler is a restricted
class. Please see the Google  App Engine developer's guide for more details.
        at
com.google.appengine.tools.development.agent.runtime.Runtime.reject(Runtime.java
:51)
        at
org.apache.commons.vfs.provider.DefaultURLStreamHandler.<init>(DefaultURLStreamH
andler.java:47)
        at
org.apache.commons.vfs.provider.AbstractFileObject$1.run(AbstractFileObject.java
:430)
        at java.security.AccessController.doPrivileged(Native Method)
        at
org.apache.commons.vfs.provider.AbstractFileObject.getURL(AbstractFileObject.jav
a:426)
        at com.newatlanta.appengine.nio.file.GaePath.toUri(GaePath.java:639)
        at
com.newatlanta.h2.store.fs.FileSystemGae.accepts(FileSystemGae.java:61)
        at org.h2.store.fs.FileSystem.getInstance(FileSystem.java:73)
        at org.h2.util.FileUtils.normalize(FileUtils.java:128)
        at org.h2.engine.ConnectionInfo.getName(ConnectionInfo.java:336)
        at org.h2.engine.Engine.openSession(Engine.java:44)
        at org.h2.engine.Engine.openSession(Engine.java:140)
        at org.h2.engine.Engine.getSession(Engine.java:120)
        at
org.h2.engine.SessionFactoryEmbedded.createSession(SessionFactoryEmbedded.java:1
7)
        at
org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:245)
        at org.h2.engine.SessionRemote.createSession(SessionRemote.java:223)
        at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:110)
        at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:94)
        at org.h2.Driver.connect(Driver.java:58)
        at java.sql.DriverManager.getConnection(DriverManager.java:582)
        at java.sql.DriverManager.getConnection(DriverManager.java:185)
        at org.h2.server.web.DbStarter.contextInitialized(DbStarter.java:39)
        at
org.mortbay.jetty.handler.ContextHandler.startContext(ContextHandler.java:530)
        at org.mortbay.jetty.servlet.Context.startContext(Context.java:135)
        at
org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1218)
        at
org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:500)
        at
org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:448)
        at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
        at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:117)
        at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
        at
org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:117)
        at org.mortbay.jetty.Server.doStart(Server.java:217)
        at
org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:40)
        at
com.google.appengine.tools.development.JettyContainerService.startContainer(Jett
yContainerService.java:181)
        at
com.google.appengine.tools.development.AbstractContainerService.startup(Abstract
ContainerService.java:116)
        at
com.google.appengine.tools.development.DevAppServerImpl.start(DevAppServerImpl.j
ava:217)
        at
com.google.appengine.tools.development.DevAppServerMain$StartAction.apply(DevApp
ServerMain.java:162)
        at
com.google.appengine.tools.util.Parser$ParseResult.applyArgs(Parser.java:48)
        at
com.google.appengine.tools.development.DevAppServerMain.<init>(DevAppServerMain.
java:113)
        at
com.google.appengine.tools.development.DevAppServerMain.main(DevAppServerMain.ja
va:89)

Original issue reported on code.google.com by [email protected] on 19 Oct 2009 at 12:44

distributed locks

A distributed locking mechanism needs to be implemented for GaeVFS. It should 
be possible to base these on memcached counters.

Original issue reported on code.google.com by [email protected] on 14 Aug 2009 at 9:11

NPE within AbstractFileObject.finalize()

Investigate NullPointerExceptions within AbstractFileObject.finalize() due 
to fs attribute being null. Are these caused by serialization changes?

Original issue reported on code.google.com by [email protected] on 14 Jul 2009 at 1:59

invoking updateContentSize failed with error:argument type mismatch

What steps will reproduce the problem?
<?php
import com.newatlanta.commons.vfs.provider.gae.GaeVFS;
import org.apache.commons.io.IOUtils;
import java.lang.Long;
import java.lang.Boolean;
GaeVFS::setRootPath(quercus_servlet_request()->getSession(true)->getServletConte
xt()->getRealPath('/'));
define('VFSM', GaeVFS::getManager());

//VFSM->resolveFile('gae://gaevfs')->createFolder();
$file=VFSM->resolveFile('gae://gaevfs/tmp1');
//$file->createFile();

$text='pp';
$method=$file->getClass()->getDeclaredMethod('updateContentSize', 
array(Long::TYPE, Boolean::TYPE));
$method->setAccessible(true);
$method->invoke($file, strlen($text), true); //error:argument type mismatch
$out=$file->getContent()->getOutputStream();
IOUtils::write($text, $out, 'UTF8');
$out->close();
?>
the noted error line raise error:argument type mismatch

What is the expected output? What do you see instead?
no error, and updateContentSize successfully

What version of the product are you using? On what operating system?
gaevfs-0.3.jar

Original issue reported on code.google.com by [email protected] on 4 Nov 2011 at 10:11

write-behind cache

A write-behind cache based on memcached and Task Queues needs to be 
implemented.

Original issue reported on code.google.com by [email protected] on 14 Aug 2009 at 9:13

enable cross jvm file locks via datastore as said was missing

enable cross jvm file locks via datastore as said was missing

i.e. a cross jvm file lock can be created via datastore. could be as simple as 
having a locked 
property on the entity. acquiring the lock would change locked property from 
false to true
in a transaction and would require an initial query similar to get file where 
key = ? and locked = 
false then set locked you true. this is similar to how object versioning is 
working and is atomic. 
should get a concurrent exception if multi attempts to lock same file at same 
time. with that said 
may make sense to maintain a version in the file which is always incremented on 
write

Original issue reported on code.google.com by [email protected] on 15 Mar 2010 at 7:50

Add a servlet filter to clear the cache as said is necessary

Add a servlet filter to clear the cache as said is necessary

i.e. write a simple filter that people can use

Original issue reported on code.google.com by [email protected] on 15 Mar 2010 at 7:35

initialize GaeVFS using "user.dir" system property

Currently, GaeVFS needs to be initialized via the GaeVFS.setRootPath() 
method. However, GAE makes the webapp root path available via the "user.dir" 
system property, so GaeVFS should be able to initialize itself if the root 
path isn't explicitly set. 

(Note, this won't work in non-GAE servlet environments).

Original issue reported on code.google.com by [email protected] on 14 Aug 2009 at 9:18

enhancement request: replace response.sendRedirect() with requestDispatcher.forward()

It would be nice to replace the response.sendRedirect() in the 
GaeVfsServlet with requestDispatcher.forward() this would make it
simpler to pass state for subsequent post-upload processing ...

Original issue reported on code.google.com by [email protected] on 29 Oct 2009 at 7:51

GAE distributed CacheListener to replace ThreadLocal cache

If GAE ever implements a distributed CacheListener, so that all running
instances of an application are notified of memcache changes, then the
GaeMemcacheFileCache ThreadLocal cache can be replaced with a more
permanent cache that's updated based on CacheListener events.

See this thread:

http://groups.google.com/group/google-appengine-java/browse_thread/thread/542f7b
44cb3d6369/faf49ee58fc35b9c?lnk=gst&q=cachelistener#faf49ee58fc35b9c

Original issue reported on code.google.com by [email protected] on 23 Jun 2009 at 8:22

batch puts of block entities

When creating a new file (such as when uploading), the current 
implementation puts each block entity to the datastore as the block 
becomes full. It might improve performance--though take more memory--to do 
a batch put of all blocks only after the file is completely written. This 
may also help with error handling and recovery when a file has only been 
partially written and a failure occurs.

Similarly, when a file is open for read/write random access, we should 
track which blocks are modified and only put those blocks in a batch when 
the file is closed. Currently, each block is put to the datastore 
individually.

Original issue reported on code.google.com by [email protected] on 21 Jul 2009 at 5:52

Add support for InputStream mark/reset and Attributes on FileObjects

I've attached two patches that I have found useful while integrating gaevfs
with the Milton WebDAV server. One adds support for mark/reset on the
FileObject's random-access InputStream; another implements the Attributes
capability on the FileObject through an extra list of properties appended
to the DataStore entity.

Original issue reported on code.google.com by [email protected] on 7 Nov 2009 at 6:52

Attachments:

• marksupport.patch
• attributesupport.patch

Virtual file size limit of 1.0MB

Due to limits on GAE datastore entity size, the maximum file that can be
written to the GaeVFS virtual file system is 1.0MB. An entity chaining
mechanism needs to be implemented to remove this limit. It's probably best
to implement this in custom InputStream and OutputStream classes. It's
important that it not be necessary to have the entire file (>1.0MB)
resident in memory; for example, it should be possible to write a file to
the servlet response output stream without ever having the entire file in
memory.

Review this for possible code sharing:
http://code.google.com/p/google-file-service/

Original issue reported on code.google.com by [email protected] on 23 Jun 2009 at 8:16

with Combined Local option, FileName.isAncestor() and FileName.isDescendant() can return unexpected results

The implementations of FileName.isAncestor() and FileName.isDescendant() 
expect the two fils to be within the same file system. With the Combined 
Local option enabled (which it is by default), this can produce unexpected 
results. For example, FileObject.getChildren() will return both local and 
GAE children for a GAE folder, but isAncestor() and isDescendant() will 
return false.

Original issue reported on code.google.com by [email protected] on 24 Jul 2009 at 9:17

junit tests

Add junit tests.

Original issue reported on code.google.com by [email protected] on 14 Aug 2009 at 9:08

use compression to reduce Entity size in datastore

Compress entities when writing to the datastore to save space. (Don't 
compress entities stored in memcache).

Original issue reported on code.google.com by [email protected] on 13 Aug 2009 at 12:54