gwadvnet2015 / gw-kv-store Goto Github PK
View Code? Open in Web Editor NEWIn-class final project: Key Value Store
License: MIT License
In-class final project: Key Value Store
License: MIT License
We plan to finish integration of the api into our code. Once the demarhsall client function is working we will integrate into our code and test it on a memcached server. This will complete the client.
What I'd like is to have these files in the client directory:
client-api.c
- this contains the functions which could be used by several different clients
client-cmd.c
- this is a command line-based client that reads arguments from the command line about whether to do a set or get and uses the library to make the request
client-perf.c
- this is a performance tester. It should issue N
requests (a command line parameter) and should measure the time taken by each request and store it in a histogram. Remember that each request will require creating a new socket connection. At the end of the program it should print out the histogram.
client-api.c
is merged with final function names.client-tests.c
- this is a test program that can be used to test whether the server is working properly. It might for example might set key1
and then set key2
and then try to get
them back to verify the values are correct. Then it might try to get key3
(which should not exist) and handle that appropriately. If the keys don't match or if the program segfaults that means the test fail. It may not be possible to run this as part of an automated Travis test because it will require that a server is actually running. That's OK with me.
I don't think we need other automated tests for the client API since all of the interesting logic is related to sockets, and there is no easy way to test that with Travis.
Write wrapper functions for the hashtable in the samples repo.
You can see the interface in this file
Development is happening here
@SMonaghan is working on implementing a MD5 hash function
@Chryel is working on the node comparison function
Extend the simple server #9 to support multiple threads. In this case, all threads should be created at program start up and they should continuously poll to see if there is a new client socket with a request that must be handled.
Write functions to parse a request and call the required function.
Extend the simple server #9 to support multiple threads (one created per client)
Write a client workload generator that can be used for evaluating overall performance. This needs to be high performance enough to stress the server. It should create N threads and have each repeatedly open a socket to the server and send a request.
It should record statistics for each thread - the response times of each request and the number of failed requests. Use the histogram and timer libraries #6 and #7.
When a key is not found, the server should only return END.
This version of the client load generator should be similar to #4, but instead of using multiple threads it should use the select
call to have a single thread monitor several different sockets. This can then be extended to support multiple threads, each using select.
How should we structure the server? How do we store data? How do we support concurrency? How should we define our application-level protocol?
It breaks.
in server/test
you should be able to run make
and have the compiled file run without error
Do it
Enabled automated continuous integration unit testing through http://travis-ci.org
See an example
Define remaining Issues - each team must clear out old issues and create new ones with specific plans to complete before next class. Your issue should not just say "finish the project", it should say specifically which parts you will have done before class--make it something reasonable that you can actually finish this week. Due Tuesday at noon.
When your team has created the issue(s), check off your tick box here--this will alert us that we can begin looking through and commenting on your plans. I've created Labels so we can tag the issues by team.
Comment on Issues - team members should indicate in comments which issues they will solve. Members from other teams should comment on other Issues to indicate which pieces are most important to them or list functions which they cannot make progress without. Due Wednesday at noon.
Write Some Code! - each team should work on their code. Team leaders should continually bug their team members to ensure progress is being made. Ideally you should do small commits and make pull requests for each small feature you add--if your pull request includes hundreds of lines of code changed it will probably be rejected and you will have to redo your work! Pull requests are due Saturday at noon.
Pull Request Reviews - Phil and Eric get the lovely job of looking through the pull requests submitted by each team. They will undoubtedly send you feedback about what should be fixed. Continue to update your code in response to their suggestions/rants. Due Monday at noon.
Think of this project as your final exam. Your final grade will be based in part on how the overall kv-store comes together and on the individual pieces you contributed. That is why it is important that you make it clear who is working on what!
@pradeep-k @hantasm @kstats @Atryus @phil-lopreiato -- you are authorized to check off boxes in this issue as your teams finish things. If you need feedback or have questions, post comments a well (I don't think we get any notification for checked boxes).
@huyang1988 @Grace-Liu @akorzan
We need to finish the Marshalling/Demarshalling stuff.
Here is the summary that we did:
1st and 4th API: Done by Yang.
2nd API done by Pradeep
3rd API is being developed by Anthony
We need to finish this as per plan.
Division of the work:
Grace: Please make the code compilable. You can see there is a Makefile and main.c inside lib directory speicifically designed for marshalling/demarshalling. Write a small test function in main.c so that we are able to call any of the APIs for testing. You don't have to write in-depth testing framework except making it run. Then, write one testcases for each of 4 APIs at least. You can write more.
Yang, pradeep and Anthony will then test their apis writing more testing code in main.c or a new file mimicing your written code as per their wish.
I will be merging the repository to main repository by end of day today so that you can fork the main repository.
We will work to finish converting our simple working string based client to a fully functioning API client. This involves fixing how we parse the incoming message and using demarshal for read response.
@akorzan has to complete this gwkv_demarshal_server
@huyang1988 has to fix the bug in gwkv_marshal_server. I believe gwkv_demarshal_client is done. If not flag it asap.
@pradeep-k has finished gwkv_marshal_client.
@Grace-Liu has finished writing the testcase. If you haven't created the pull request, please do this asap.
Since, other teams are depending on these APIs, we need to make this up first. I am targeting it to be finished by Wednesday 11:59 PM.
@akorzan and @huyang1988 : Let us know if you guys need any help in finishing this by that time.
Will attempt to complete the timer utility's last timer_end function with histogram features.
We use thread pool. There is a task list. The procedure is the following:
Need other groups to provide the parse_message and process_operation functions for us.
void parse_message(char *message, int len, struct operation *operation);
void process_operation(struct operation *operation, char *message, int *len);
Extend the server #9 so that it has a single thread that calls select
to multiplex multiple client sockets.
Make a wrapper around the hash table library so that it can be easily used to support get
, set
, and inc
operations.
https://github.com/gwAdvNet2015/adv-net-samples/tree/master/datastructs/hashtable
Finish this and import it to this project so we can use it in our client:
gwAdvNet2015/adv-net-samples#8
Write a simple client that issues N get or set requests. Test it using a standard memcached server.
all files should begin with:
/************************************************
* GW KV
* https://github.com/gwAdvNet2015/gw-kv-store
*
* Copyright 2015 Author 1, Author 2,
* Author 3, Author 4
*
* This program is licensed under the MIT license.
*
* filename.c - parses incoming messages and
* and prints them to the screen.
*************************************************/
Replace the authors with anyone who worked on the file and put a short description of what the file does.
Be sure this works and import it so we can use it in the client libraries
gwAdvNet2015/adv-net-samples#12
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.