apereo-learning-analytics-initiative/openlrw

Standards-based learning record warehouse built for a scalable learning analytics environment.

Apereo OpenLRW

OpenLRW is a Java-based learning record warehouse which is compatible with xAPI (Experience API), IMS Caliper, and IMS OneRoster. OpenLRW is a secure, standards-based, standalone learning record warehouse that was built to fill the need for a storage mechanism for an open learning analytics environment. Built on a scalable architecture, using modern web technologies, Apereo LRW provides the fast reads and writes necessary for a dynamic analytics environment.

Documentation

The documentation is available by clicking on this link.

License

OpenLRW is made available under the terms of the Educational Community License, Version 2.0 (ECL-2.0).

Contact

Send questions or comments to the mailing list: [email protected]

openlrw's Issues

a

Project needs a LICENSE file.

Apache 2.0 preferred; ECL only if necessary.

About how to initialize mongodb instance correctly

Hi, i am having problems to run the jar file with a mongo server running within a docker container.

This is my application.yaml file:

`# different mongo instance running within a docker container
spring.data.mongodb.host: 172.17.0.3
spring.data.mongodb.port: 27017

server.port: 9966
spring.profiles: default
matthews.security.jwt:
tokenExpirationTime: 240 # Number of minutes
refreshTokenExpTime: 360 # Minutes
tokenIssuer: http://example.com
tokenSigningKey: changeme

Spring Actuator

See https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html

Disable by default all but info

endpoints:
enabled: false
info:
enabled: true

info:
build:
artifact: @project.artifactId@
name: @project.name@
description: @project.description@
version: @project.version@

matthews.users:
encrypted: false
adminuser : ${random.value}
password: ${random.value}
emailAddress: ${random.value}@unicon.net`

This is an output of my docker image running locally:

`...
2017-09-14T13:20:00.127+0000 [clientcursormon] mapped:64
2017-09-14T13:20:00.127+0000 [clientcursormon] connections:1
2017-09-14T13:24:30.926+0000 [initandlisten] connection accepted from 172.17.0.1:55338 #2 (2 connections now open)
2017-09-14T13:24:31.479+0000 [initandlisten] connection accepted from 172.17.0.1:55340 #3 (3 connections now open)
2017-09-14T13:24:31.504+0000 [conn3] assertion 13 not authorized for query on test.tenant ns:test.tenant query:{ name: "DEFAULT_TENANT" }
2017-09-14T13:24:31.504+0000 [conn3] ntoskip:0 ntoreturn:-1
2017-09-14T13:24:31.579+0000 [conn3] end connection 172.17.0.1:55340 (2 connections now open)
2017-09-14T13:24:31.580+0000 [conn2] end connection 172.17.0.1:55338 (1 connection now open)
2017-09-14T13:25:00.137+0000 [clientcursormon] mem (MB) res:66 virt:212
2017-09-14T13:25:00.137+0000 [clientcursormon] mapped:64
2017-09-14T13:25:00.137+0000 [clientcursormon] connections:1

exit
bye
2017-09-14T13:25:52.025+0000 [conn1] end connection 127.0.0.1:47006 (0 connections now open)
WHAT IS MY IP?
root@d97d64355595:/# ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:ac:11:00:03
inet addr:172.17.0.3 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:47 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:5978 (5.9 KB) TX bytes:3315 (3.3 KB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:139 errors:0 dropped:0 overruns:0 frame:0
TX packets:139 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1
RX bytes:17541 (17.5 KB) TX bytes:17541 (17.5 KB)

root@d97d64355595:/#
`

And i am getting this exception when i run the project:

Caused by: com.mongodb.MongoQueryException: Query failed with error code 13 and error message 'not authorized for query on test.tenant' on server 172.17.0.3:27017 at com.mongodb.connection.ProtocolHelper.getQueryFailureException(ProtocolHelper.java:131) at com.mongodb.connection.QueryProtocol.execute(QueryProtocol.java:303) at com.mongodb.connection.QueryProtocol.execute(QueryProtocol.java:54) at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:159) at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:286) at com.mongodb.connection.DefaultServerConnection.query(DefaultServerConnection.java:209) at com.mongodb.operation.FindOperation$1.call(FindOperation.java:496) at com.mongodb.operation.FindOperation$1.call(FindOperation.java:482) at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:239) at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:212) at com.mongodb.operation.FindOperation.execute(FindOperation.java:482) at com.mongodb.operation.FindOperation.execute(FindOperation.java:79) at com.mongodb.Mongo.execute(Mongo.java:772) at com.mongodb.Mongo$2.execute(Mongo.java:759) at com.mongodb.DBCollection.findOne(DBCollection.java:777) at com.mongodb.DBCollection.findOne(DBCollection.java:747) at com.mongodb.DBCollection.findOne(DBCollection.java:694) at org.springframework.data.mongodb.core.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:2143) at org.springframework.data.mongodb.core.MongoTemplate$FindOneCallback.doInCollection(MongoTemplate.java:2127) at org.springframework.data.mongodb.core.MongoTemplate.executeFindOneInternal(MongoTemplate.java:1901) ... 165 more
I guess that another process needs to push some initial data to the mongo instance, isnt?
My question is, how do i do that initial push?

Slow Startup on Digital Ocean

I installed OpenLRW and it took over 20 minutes to start up on Digital Ocean on a 2CPU/4GB Ubuntu server. Turns out there is an "entropy problem" - here are my notes for the fix:

apt-get install haveged
update-rc.d haveged defaults
apt-get install rng-tools
cat /dev/random | rngtest -c 1000

• http://www.issihosts.com/haveged/
• https://www.digitalocean.com/community/tutorials/how-to-setup-additional-entropy-for-cloud-servers-using-haveged
• https://www.digitalocean.com/community/questions/fresh-tomcat-takes-loong-time-to-start-up
• https://stackoverflow.com/questions/27612209/spring-boot-application-wont-boot-at-startup-inside-docker

Integrating with Articulate xAPI content

I would like to integrate with Articulate content.
It is my understanding that following params are required to connect to LRS

http://my.lms.com/TCActivityProvider/story.html
?endpoint=http://my.lms.com/lrs/endpoint/
&auth=Basic OjFjMGY4NTYxNzUwOGI4YWY0NjFkNzU5MWUxMzE1ZGQ1
&actor={"name": ["First Last"], "mbox": ["mailto:[email protected]"]}
&activity_id=61XkSYC1ht2_course_id
&registration=760e3480-ba55-4991-94b0-01820dbd23a2

• What would be the endpoint for this LRS to connect
• For admin login I can get the access token (JWT token), what would be required to passed in auth param
• What would be in registration value?

This is screenshot when export option is available in articulate

Thanks.

Caliper - Can not construct instance of unicon.matthews.caliper.Agent$Builder

I'm getting this error:

{"status":"BAD_REQUEST","messages":["Can not construct instance of unicon.matthews.caliper.Agent$Builder: no String-argument constructor/factory method to deserialize from String value (''https://platform.example.org'')"],"path":"/api/caliper","timestamp":"2018-08-10T19:02:53.382"}

Sending the following as POST w valid credentials:
https://gist.github.com/jrissler/d308d5a78049151dc5ffa296f9a7d5fa - not sure whats up?

feature: route GET /lineitems

A route for getting all the line items

Flow of the project

Hi,

I saw your interesting new commits but it removed some of my recent contributions.
Did my contributions not fit with the flow of the project?

Regards

xAPI to Caliper conversion: verb URIs

Hi,

I'm currently adding xAPI events through Logstash scripts and we are facing an issue.

In the xAPI JSON structure, there is a field id for the verb object such as :

"verb": {
         "id": "https://w3id.org/xapi/adl/verbs/logged-in",
          "display": {
            "en-US": "Logged in",
          }
}

At first, I googled websites that have an xAPI dictionary (I found w3id.org which is pretty nice) however I noticed the verb id isn't converted to Caliper; indeed this URI hasn't been added in DefaultXapiToCaliperConversionService.java

My question is why are there so many different websites for verbs ? (tincanapi.com, brindlewaye.com, activitystrea.ms, etc...)

Each time we have to check the java file to know which hostname has been picked for this verb.

Can I fix/pull request this and use the same hostname for all the verbs, or something like this ?

Thanks

Unsupported field: InstantSeconds When Receiving statements

Hi,
I was successfully able to deploy OpenLRW, Post an xAPI statement:

{

  "actor": {
    "name": "Sally Glider",
    "mbox": "mailto:[email protected]"
  },
  "verb": {
    "id": "http://adlnet.gov/expapi/verbs/experienced",
    "display": { "en-US": "experienced" }
  },
  "object": {
    "id": "http://example.com/activities/solo-hang-gliding",
    "definition": {
      "name": { "en-US": "Solo Hang Gliding" }
    }
  }
}

I also verified that it is stored in the MongoDB. When I try to receive all the statements or one statement from the endpoint GET xAPI/statements I get the following error, as per the logs:

java.time.temporal.UnsupportedTemporalTypeException: Unsupported field: InstantSeconds
at java.time.LocalDate.get0(LocalDate.java:680)
at java.time.LocalDate.getLong(LocalDate.java:659)
at java.time.LocalDateTime.getLong(LocalDateTime.java:720)
at java.time.format.DateTimePrintContext.getValue(DateTimePrintContext.java:298)
at java.time.format.DateTimeFormatterBuilder$InstantPrinterParser.format(DateTimeFormatterBuilder.java:3178)
at java.time.format.DateTimeFormatterBuilder$CompositePrinterParser.format(DateTimeFormatterBuilder.java:2179)
at java.time.format.DateTimeFormatter.formatTo(DateTimeFormatter.java:1746)
at java.time.format.DateTimeFormatter.format(DateTimeFormatter.java:1720)
at java.time.LocalDateTime.format(LocalDateTime.java:1752)
at unicon.matthews.xapi.service.DefaultXapiToCaliperConversionService.toXapi(DefaultXapiToCaliperConversionService.java:610)
at unicon.matthews.xapi.endpoint.XapiApiController.getStatements(XapiApiController.java:139)

I think it is in the code: DefaultXapiToCaliperConversionService.java:610, some error during parsing the date. In the db the date is stored as "eventTime" : ISODate("2017-06-20T06:57:01.011Z") .

Any solutions for this?

Configuration Issues

Hello,
I am working with OpenLRW and am trying to set it up on my host machine
where I have my MongoBD. I have confirmed that it does connect with my
mongoDB. I see that in the github updates you mentioned a UI however when I
go to localhost:9966 i get a whitlelabel error saying there is no explicit
mapping. is this correct or is there something wrong with my configuration?
or is this just the endpoint to send data to with no ui created?

where is OpenLRW client code?

i can't find OpenLRW Client in this repository.how can i find it ? thank you!

Adding an API call for all courses and all users

Would it be possible to get a list of all courses and all users stored in the LRW thanks to the API ?
A kind of GET request like /api/classes/all and /api/users/all would be nice.

Thanks,

Java 9 Support

OpenLRW fails to start when using Java 9.

Mongodb URI

Please provide format for mongodb url for run.sh file ....

#!/bin/sh
cd dirname $0
APP_HOME="$PWD"
PID_FILE=$APP_HOME/run/openlrw.pid
JAR_PATH=$APP_HOME/lib/openlrw.jar

cd $APP_HOME

case "$1" in
"start")
if [ -f $PID_FILE ]; then
exit 1
fi
java
-Dlogging.path=/opt/openlrw/logs/
-Dspring.data.mongodb.uri= <!-- ###mongodb uri -->
-jar $JAR_PATH &
echo $! > $PID_FILE
;;
"stop")
if [ ! -f $PID_FILE ]; then
exit 1
fi
kill cat $PID_FILE
rm -f $PID_FILE
;;
*)
echo "Usage: $0 start|stop"
;;
esac
exit 0

JSON statement from Sakai failing

When grading an assignment, the SakaiXAPI provider generates this json below and gets a 500 error back from OpenLRW. Other statements generated from this provider are accepted. This previously worked in OpenLRS. I'm not sure where the error is.

Stacktrace from OpenLRW

Caused by: java.lang.IllegalStateException: unicon.matthews.caliper.Agent@7c5a2157[
  id=<null>
  type=https://github.com/adlnet/xAPI-Spec/blob/master/xAPI.md#agentaccount
  context=http://purl.imsglobal.org/ctx/caliper/v1/Context
  name=John Fitz Gerald
  description=<null>
  extensions={}
  dateCreated=<null>
  dateModified=<null>
]
	at unicon.matthews.caliper.Agent$Builder.build(Agent.java:216)
	at unicon.matthews.xapi.service.DefaultXapiToCaliperConversionService.fromXapi(DefaultXapiToCaliperConversionService.java:272)
	at unicon.matthews.xapi.endpoint.XapiApiController.postStatement(XapiApiController.java:93)
	... 117 common frames omitted

JSON from Sakai

{
  "actor": {
    "name": "John Fitz Gerald",
    "account": {
      "name": "",
      "homePage": ""
    },
    "objectType": "Agent"
  },
  "result": {
    "completion": true,
    "score": {
      "min": 0.0,
      "max": 111.0,
      "raw": 10.0
    }
  },
  "verb": {
    "id": "http:\/\/www.adlnet.gov\/expapi\/verbs\/scored"
  },
  "context": {
    "contextActivities": {
      "other": {
        "id": "http:\/\/adlnet.gov\/expapi\/activities\/assignment"
      }
    },
    "instructor": {
      "name": "The Instructor",
      "account": {
        "name": "instructor",
        "homePage": "http:\/\/localhost"
      },
      "objectType": "Agent"
    }
  },
  "object": {
    "definition": {
      "name": {
        "en-US": "User received a grade"
      },
      "description": {
        "en-US": "User received a grade for their assginment: Single File Only; Submission #: 0"
      },
      "type": "http:\/\/adlnet.gov\/expapi\/activities\/received-grade-assignment"
    },
    "id": "http:\/\/localhost\/portal\/assignment\/s\/87768560-45c3-4aa7-9a29-d012384fec11\/26c0b7e8-a08f-4f9e-96b7-ceb13c063a26\/b9dac0f7-a0ae-4779-b1c7-547fc2895b5b",
    "objectType": "Activity"
  },
  "timestamp": "2017-11-07T15:44:59-05:00"
}

Remove classSourcedId in Result

Result objects contain a field named "classSourcedId", there is a problem about this.

Context

ClassSourcedId is an attribute of the LineItem objects.

Following the OneRoster structure, a Result is linked to a LineItem and a User.

When creating a Result object, the API expects you to use the following HTTP POST route /classes/:id/results

The value of classSourcedId is gotten by using the URI and then added to the Result object.

Structure issue

First, the used route is already not "OneRoster compatible" since there is no links between Results and Classes (it should have been something like /lineitems/:id/results )

The classSourcedId value is already supposed to be in the LineItem object (following the OneRoster structure).

Production issue

If you PATCH a LineItem and you update its classSourcedId it is not gonna update all the Result objects that are linked to this LineItems, which means this value is going to be wrong for all these objects.

What do you think about it?

Consider renaming packages from unicon to org.apereo

Apereo matters. :)

Post data

Hi ,

I have Question related POST request ?
Can we send data through Authorization with username and password.
Because large data transfer can required to update API Key many times.
Token Reset regularly .

1. Can we fixed API key value (Not change automatically)
2. Code for POST data through Username and Password instead of API key.
   Thanks

Bad Request(400) is returned when using "caliper-common-fixtures-public"

I try to use the following test fixture to confirm whether it works or not.
https://github.com/IMSGlobal/caliper-common-fixtures-public/blob/public/src/test/resources/fixtures/caliperSessionLoginEvent.json

However, when I POST this fixture to OpenLRW, it emits the next error.

{'messages': ['Can not deserialize instance of java.lang.String out of '
              'START_OBJECT token'],
 'method': 'POST',
 'parameters': {},
 'path': '/key/caliper',
 'reason': 'Bad Request',
 'status': 400}

If I omit the actor object in generated section from the fixture, it does work.
It seems to return correctly and is saved to MongoDB.

['1fb29429-47c8-4aee-8741-49e8aa51638c']

Which does this framework support Caliper, overall or partially?
If it has overall support, I suspect this is a bug.

I'm sorry but I'm so new to Java that I cannot give a test code which reproduce the symptom.

xAPI to Caliper conversion

Hello,

I am currently looking for an LRS to run locally to save xAPI statements.I have seen in this LRW every xAPI statement received is converted to Caliper format and then saved to LRS.Can we save the statements in xAPI format directly instead of converting them to Caliper format and then saving?

org.apereo.openlrw.caliper.*

some java file import classes from a org.apereo.openlrw.caliper package, but this is not in the src tree.
where can i find them ?

Session Sensor Error message

I have several sensors written for Navigation, Link click events that work correctly. However, when I run this session event, I get an error response. This seems to be related to json deserializing the actor entity that is added to the generatedsession.

Caliper JSON:

{
"sensor": "id",
"sendTime": "2017-11-08T17:04:34.000Z",
"data": [
{
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
"@type": "http://purl.imsglobal.org/caliper/v1/SessionEvent",
"actor": {
"@id": "[email protected]",
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
"@type": "http://purl.imsglobal.org/caliper/v1/lis/Person",
"name": null,
"description": null,
"extensions": {},
"dateCreated": null,
"dateModified": null
},
"action": "http://purl.imsglobal.org/vocab/caliper/v1/action#LoggedIn",
"object": {
"@id": "https://courses.worldcampus.psu.edu/",
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
"@type": "http://purl.imsglobal.org/caliper/v1/SoftwareApplication",
"name": "Penn State worldcampus",
"description": null,
"extensions": {},
"dateCreated": null,
"dateModified": "2017-11-08T17:04:34.000Z"
},
"target": null,
"generated": {
"@id": "http://imsglobal.org/sampleCaliperApp/session-123456789",
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
"@type": "http://purl.imsglobal.org/caliper/v1/Session",
"name": "session-123456789",
"description": "new",
"extensions": {},
"dateCreated": "2017-11-08T17:04:34.000Z",
"dateModified": null,
"actor": {
"@id": "[email protected]",
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
"@type": "http://purl.imsglobal.org/caliper/v1/lis/Person",
"name": null,
"description": null,
"extensions": {},
"dateCreated": null,
"dateModified": null
},
"startedAtTime": "2017-11-08T17:04:34.000Z",
"endedAtTime": "2017-08-16T05:00:00.000Z",
"duration": "PT22S"
},
"eventTime": "2017-11-08T17:04:34.000Z",
"edApp": {
"@id": "https://courses.worldcampus.psu.edu/",
"@context": "http://purl.imsglobal.org/ctx/caliper/v1/Context",
"@type": "http://purl.imsglobal.org/caliper/v1/SoftwareApplication",
"name": "Penn State worldcampus",
"description": null,
"extensions": {},
"dateCreated": null,
"dateModified": "2017-11-08T17:04:34.000Z"
},
"group": null,
"membership": null,
"federatedSession": null
}
]
}

Response
{"description":"Could not read document: Can not deserialize instance of java.lang.String out of START_OBJECT
token\n at [Source: java.io.PushbackInputStream@347b5315; line: 39, column: 26] (through reference chain
: unicon.matthews.caliper.Builder["data"]->java.util.ArrayList[0]->unicon.matthews.caliper.Builder
["generated"]->unicon.matthews.caliper.Builder["actor"]); nested exception is com.fasterxml.jackson
.databind.JsonMappingException: Can not deserialize instance of java.lang.String out of START_OBJECT
token\n at [Source: java.io.PushbackInputStream@347b5315; line: 39, column: 26] (through reference chain
: unicon.matthews.caliper.Builder["data"]->java.util.ArrayList[0]->unicon.matthews.caliper.Builder
["generated"]->unicon.matthews.caliper.Builder["actor"])"}
);</script>

Multiple JSON Object send through Post

How to send Multiple Object through Rest POST method.
Actually I have create an array but It propagate Message..

{
"status": "BAD_REQUEST",
"messages": [
"Can not deserialize instance of unicon.matthews.caliper.Envelope$Builder out of START_ARRAY token"
],
"path": "/api/caliper/",

Swagger does not work anymore

I get an ERR_EMPTY_RESPONSE when I check the website

Converting UTC time sent in Caliper Event again as it local time sent

We sending caliper events to an openLRW setup on our server. We are sending the sensor sendTime, and the event eventTime in UTC format, but the openLRW is again converting the time as if it were local time.

Ex. the event.eventTime field sent to openLRW is "2018-12-04T15:29:22Z"
but what gets stored in the mongo Event for event.eventTime is ISODate("2018-12-04T20:29:22Z")

This is a big issue for us, because we are doing data analysis by local date, and this throws the data off.

I can't find in the code where this conversion is taking place.

Proper way to add scripts and credentials file to the repository

Hi,

This repository presents some little issues imo:

• there isn't a settings file for the MongoDB credentials
• Shell scripts and folders are not physically featured (people have to copy and past from the README file)

Therefore the UX is perfectible.

In my mind, I rather see the OpenLRW project as a whole package including the Java API and other services (shell scripts and other stuff; like in skeleton projects) than a "stand-alone" API without some useful files.

I deeply think that including the above elements would make the installation easier and faster (cloning the project and vi the credentials file is much easier and safer than mkdir folders and creating/copying scripts from the README file on the github web interface);

I would like to discuss with you about your feelings about this and about how to provide these elements to the repository and to the community.

Last time I've taken the initiative to change the structure but I didn't really well explained it; and I think I should have asked you before doing this.

Thanks for reading.

Regards,

Xavier