Metadata API current usage
Client initiated Metadata operations
The following client operations will create a META command:
- Call to
get_global_file_meta() will retrieve unifycr_fattr_t from the server (flag/type is 1).
- Call to
set_global_file_meta() will send serialized unifycr_fattr_t to the server, which stores it in MDHIM(flag/type is 2).
- Call to
fsync() on the client will send a "fsync" message to the server (flag/type is 3).
Call to unifycr_fd_logreadlist() will create "READ" command for the server.
The functions set_global_file_meta() and get_global_file_meta() are called from unifycr_fid_open().
Synchronization to Delegator
In unifycr_sync_to_del(), the client side information is transferred to the server. The unifycr_sync_to_del() function is called from unifycr_mount() (Metadata operation is COMM_MOUNT).
Server Metadata ops
The server handles Metadata operations initiated by the client in the delegator_handle_command() function (unifycr_cmd_handler.c). Depending on the operation, meta_process_fsync(), meta_process_attr_set(), meta_process_attr_get() are called.
Note: the sub commands are not defined in an enum.
From request manager rm_read_remote_data() call to meta_batch_get(). The request manager dispatches read requests to different threads based on the requesting client.
meta_process_attr_set() stores file attributes into mdhim.
meta_process_attr_get() retrives file attributes from mdhim.
meta_process_fsync() stores all file attributes and extents in mdhim.
meta_batch_get() retrieves extents from mdhim.
superblock
The superblock is allocated in in the client in unifycr_superblock_shmget(). If the superblock already exists, the client will attach to the existing superblock.
Pointers into the superblock are set up in unifycr_init_pointers(). The following data lives in the superblock:
- a header (uint32_t)
free_fid_stackunifycr_filelistunifycr_filelistunifycr_chunkmetas- spillover padding?
free_chunk_stack- spillover padding
unifycr_chunks (only if unifycr_use_memfs if set)- if
fs_type is UNIFYCR_LOG
unifycr_indices.ptr_num_entriesunifycr_indices.index_entryunifycr_fattrs.ptr_num_entriesunifycr_fattrs.meta_entry
Metadata
File attributes
The server defines the following type in unifycr_global.h:
typedef int fattr_key_t;
typedef struct {
char fname[ULFS_MAX_FILENAME];
struct stat file_attr;
} fattr_val_t;
typedef struct {
int fid;
int gfid;
char filename[ULFS_MAX_FILENAME];
struct stat file_attr;
} unifycr_file_attr_t;
fattr_val_t is set using meta_process_attr_set() and retrieved using meta_process_attr_get(). In meta_process_fsync() fattr_val_t and unifycr_val_t are exchanged. All unifycr_val_t are retrieved with a call to meta_batch_get().
The client defines the following type in unifycr_internal.h:
typedef struct {
int fid;
int gfid;
char filename[UNIFYCR_MAX_FILENAME];
struct stat file_attr;
} unifycr_fattr_t;
The types unifycr_file_attr_t and unifycr_fattr_t are identical. Client and server use the unifycr_f(ile_)attr_t type to exchange the file attributes between client and server using the socket. The server in turn uses
fattr_key_t and fattr_val_t to store the file attributes in mdhim.
The file attributes are exchanged with the client.
Types defined by client and server
typedef struct {
off_t file_pos;
off_t mem_pos;
size_t length;
int fid;
} unifycr_index_t;
The unifycr_index_t type is defined in unifycr-internal.h for the client and unifycr_metadata.h for the server.
File extents?
typedef struct {
unsigned long fid;
unsigned long offset;
} unifycr_key_t;
typedef struct {
unsigned long delegator_id;
unsigned long len;
unsigned long addr;
unsigned long app_rank_id; /*include both app and rank id*/
} unifycr_val_t;
unifycr_key_t and unifycr_val_t are set in meta_process_fsync. The data is copied from the superblock and stored in a global array for the keys and values (unifycr_keys and unifycr_vals). After the data is copied, it is stored in the metadata server (using mdhimBPut()).
File and chunk meta
The following types are defined be the client in unifycr-internal.h
typedef struct {
off_t size; /* current file size */
off_t real_size; /* real size of the file for logio*/
int is_dir; /* is this file a directory */
pthread_spinlock_t fspinlock; /* file lock variable */
enum flock_enum flock_status; /* file lock status */
int storage; /* FILE_STORAGE specifies file data management */
off_t chunks; /* number of chunks allocated to file */
unifycr_chunkmeta_t *chunk_meta; /* meta data for chunks */
} unifycr_filemeta_t;
typedef struct {
int location; /* CHUNK_LOCATION specifies how chunk is stored */
off_t id; /* physical id of chunk in its respective storage */
} unifycr_chunkmeta_t;
There is an array on the superblock to store unifycr_max_files of unifycr_filemeta_t and unifycr_max_chunks of unifycr_chunkmeta_t types.
In the client unifycr_filemeta_t is retrieved by a call to unifycr_get_meta_from_fid().
Functions reading unifycr_filemeta_t:
unifycr_fid_is_dirunifycr_fid_sizeunifycr_fid_statunifycr_fid_readunifycr_fid_write (calls unifycr_fid_store_fixed_write)unifycr_fid_shrink (calls unifycr_fid_store_fixed_shrink)unifycr_fid_store_fixed_write
Functions writing unifycr_filemeta_t:
unifycr_fd_writeunifycr_fid_store_allocunifycr_fid_create_fileunifycr_fid_create_directoryunifycr_fid_extendunifycr_fid_truncateunifycr_fid_open
open questions
Do we really need to store the entire stat structure in the K-V store?
Does the fsync need to store file attributes?
Proposed new Metadata API
shared metadata types
Datatypes shared by the client and server need to be defined in a common directory.
Common datatypes:
unifycr_file_attr_t and unifycr_fattr_t (need to rename one of the types)unifycr_index_t
proposed API functions
/*
*
*/
int unifycr_set_file_attribute (const char* const filename, struct stat file_stat);
/*
*
*/
int unifycr_get_file_attribute (const char* const filename, struct stat *file_stat);
/*
*
*/
int unifycr_set_file_extents (const char* const filename, unsigned int num_extents, unifycr_index_t *extents);
/*
*
*/
int unifycr_bulk_set_file_extents (unsigned int num_files, const char** const filename, unsigned int *num_extents, unifycr_index_t **extents);
/*
*
*/
int unifycr_get_file_extents (const char* const filename, unsigned int *num_extents, unifycr_index_t **extents);
/*
*
*/
int unifycr_bulk_get_file_extents (unsigned int num_files, const char* const filename, unsigned int **num_extents, unifycr_index_t ***extents);
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent