Communication with Neo4j graph database

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Communication with Neo4j graph database

Post by stefan » Mon Sep 05, 2016 7:52 pm

Hello,

first of all, I am very new to Odysseus. If there is a simple solution, just give me a short hint.
I want to develop a prototype for a multimodal and integrated mobility platform.
Therefore I stored several public transport stops/stations and start-/stops of car sharing, rideshare etc. in a Neo4j database. Neo4j is a graph database and provides several features to manipulate and lookup graph structures (more) easily.
Odysseus should act as platform. Users can send mobility offers and mobility requests and Odysseus will provide the corresponding information for the user. Therefore, I have to read and write from/to the Neo4j database (if necessary multiple times) during the event processing. The result of this database query shold be a new event with a subgraph in its payload. Ideally this subgraph is readable and modifiable through Odysseus.

My first idea was to use the REST-API (http://neo4j.com/docs/rest-docs/current/) and the Rapid Prototyping Feature to create an operator like "DBEnrich". But in my first opinion, the Rapid Prototyping Feature is limited because it is just possible to call one script or one single Java-Class. You cannot use more classes that interact with each other etc. It seems to be tricky to develop such a functionality in that way.

I did not actually want to create a new database interface through the Access framework or an new operator because it is just a prototype. What is the best and/or fastest way to go with Neo4j for a prototype? If the Rapid Prototyping Feature can be used - what are the steps I should pay attention?

Thanks in advance,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Tue Sep 06, 2016 9:04 am

Hello Stefan,

welcome to Odysseus :-)

Graph processing is a feature we are currently developing and its in a rather alpha state ;-).

In which way do you use Odysseus? Do you have downloaded a prebuild version oder do you use the source code?

Im an not sure, if this is possible with a Neo4J database, but you could try to use the WSEnrich operator:

http://wiki.odysseus.offis.uni-oldenbur ... h+operator

Maybe, this solves your problem? It not maybe we can adapt the operator a little bit?

Greetings,

Marco

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Tue Sep 06, 2016 11:16 am

Hello Marco,

thank you for your fast reply.
Thats great to hear! For my prototype this will be to late, but for further development this could be a very useful feature. ;)

I downloaded the newest prebuild Odysseus Server + Odysseus Studio Client for Windows x64 nightly build. Odysseus should get several information (user requests, mobility offers, weather information etc). This information should be preprocessed to query the Neo4j database in order to determine the cheapest/fastest/best connection between a point A and B. So, Odysseus is actually doing the whole processing while Neo4j is used for graph processing and to store the data as a graph. But I am at the very beginning, therefore I just evaluated some functionality in Odysseus and did not realize a specific functionality.

The WSEnrich Operator is a good hint. I did not know this operator up to now. I will check its functionality. Is it possible to call the REST Interface with URL+Header-Parameters+RequestBody as POST? If so, how can I add the body?

Thank you very much, I think, this operator could do it!

Greetings,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Wed Sep 07, 2016 8:44 am

Hello Stefan,

Hmm. I am not sure ;-)

As far as I can see, the parameters for the request are extracted from the incoming element. Currently, I have no ways to tests, but maybe you can try?

Set method to post and use the MAP-Operator before the WSEnrich to include the information that should is required in the post method. I guess this is something like an access token?

Greetings,

Marco

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Fri Sep 09, 2016 1:56 pm

Hello Marco,

you are right. I want to pass (in my currently simplified test environment) something like this:

Code: Select all

POST http://localhost:7474/db/data/node/1/paths HTTP/1.1
Authorization: Basic bmVvNGo6bmVvcGFzcw==
Host: localhost:7474

{
  "to" : "http://localhost:7474/db/data/node/5",
  "cost_property" : "cost",
  "relationships" : {
    "type" : "connected_by",
    "direction" : "out"
  },
  "algorithm" : "dijkstra"
}
i. e. the Method "Post", an URL, a header attribute for authorization and a body with the corresponding "instructions".

Therefore I created a simple CSV file with some values that I am using as a source in Odysseus. I tried different ways to pass this arguments through URL, Arguments etc. I could not figure out, how I have to use it.

To start with an easier example I tried a simple "get" for one node.

Code: Select all

neo4jResponse = WSENRICH(
	{
		SERVICEMETHOD = 'REST',
		METHOD = 'GET',
		URL = 'http://localhost:7474/db/data/node/1',
		ARGUMENTS = [['Authorization', 'Authorization']],
		PARSINGMETHOD = 'JSONPATH',
		DATAFIELDS = [
			['$.[0].name', 'String']
		]
	}, 
	src_neo4jTest
)
The csv file contains this data (I just use the Authorization param at the moment):

Code: Select all

to;Authorization;Host;cost_property;relationships;algorithm
http://localhost:7474/db/data/node/5;'Basic bmVvNGo6bmVvcGFzcw==';'localhost:7474';"cost";'{ "type" : "connected_by", "direction" : out }';"dijkstra"
At the command line, I can see this:
5061004 WARN AbstractEnrichPO - Empty result for input http://localhost:7474/db/data/node/5|Basic bmVvNGo6bmVvcGFzcw==|localhost:7474|"cost"|{ "type" : "connected_by", "direction" : out }|"dijkstra" | META | 1473420723540|oo - de.uniol.inf.is.odysseus.core.server.physicaloperator.AbstractEnrichPO.process_next(AbstractEnrichPO.java:111)

Because I dont know how the request looks like, I cannot understand why the result is empty.

To make things easier and maybe better to test for others, I used an online Rest-test service: https://jsonplaceholder.typicode.com/
I want to call the users which is done easily by: http://jsonplaceholder.typicode.com/users
To get the first name one can use this JsonPath expression: $.[0].name (tested via http://jsonpath.com/ online evaluator)
The input is the same csv file as before.

Because it is not possible to skip the Arguments section in the query, I added the Authorization part. I tested the Webservice Call (incl. Authorization) via Postman and Fiddler. The Webservice ignores this attribute and responses as expected.
My Odysseus Query looks like this:

Code: Select all

wsresponse = WSENRICH(
	{
		SERVICEMETHOD = 'REST',
		METHOD = 'GET',
		URL = 'http://jsonplaceholder.typicode.com/users',
		ARGUMENTS = [['Authorization','Authorization']],
		PARSINGMETHOD = 'JSONPATH',
		DATAFIELDS = [
			['$.[0].name', 'String']
		]
	}, 
	src_neo4jTest
)
At the Odysseus Studio Command line I can see the following:
5982445 WARN AbstractEnrichPO - Empty result for input http://localhost:7474/db/data/node/5|Basic bmVvNGo6bmVvcGFzcw==|localhost:7474|"cost"|{ "type" : "connected_by", "direction" : out }|"dijkstra" | META | 1473421644912|oo - de.uniol.inf.is.odysseus.core.server.physicaloperator.AbstractEnrichPO.process_next(AbstractEnrichPO.java:111)

Maybe its a really easy mistake but up to now I have no glue what I am doing wrong.
Can someone offer a hint? :)

Thanks!

Greetings,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Fri Sep 09, 2016 3:43 pm

Hi Stefan,

this operator is written a long time ago by a student so there seems to be not tested everything in detail ;-)

What I found out: You need to add the ? in the get request manually.

So my full example looks like this:

Code: Select all

#PARSER PQL
#RUNQUERY

src_neo4jTest = CSVFILESOURCE({
                    schema = [
                      ['to', 'String'],
                      ['Authorization','String'],
                      ['Host','String'],
                      ['cost_property','String'],
                      ['relationships','String'],
                      ['algorithm','String']
                    ],
                    source = 'testsource',
                    filename = '${PROJECTPATH}/input.csv',
                    delimiter = ";",
                    textdelimiter = '\''                                                        
                  }                                                  
                )

neo4jResponse = WSENRICH(   {
                    servicemethod = 'REST',
                    method = 'GET',
                    url = 'http://jsonplaceholder.typicode.com/users?',
                    arguments = [['Authorization', 'Authorization']],
                    parsingmethod = 'JSONPATH',
                    datafields = [
                      ['$.[0].name', 'String']
                    ]                                
                  },
                  src_neo4jTest
                )
With input.csv:

Code: Select all

http://localhost:7474/db/data/node/5;'Basic bmVvNGo6bmVvcGFzcw==';'localhost:7474';"cost";'{ "type" : "connected_by", "direction" : out }';"dijkstra"
When I run this query, I get an output:

Code: Select all

http://localhost:7474/db/data/node/5|Basic bmVvNGo6bmVvcGFzcw==|localhost:7474|"cost"|{ "type" : "connected_by", "direction" : out }|"dijkstra"|Leanne Graham | META | 1473428246406|oo
We will fix this.

I found another issue here: Restart of a query with WSENRICH seems not possible ...

BTW: The URL is printed. You can see it when you active TRACE in the logging-options:
http://wiki.odysseus.offis.uni-oldenbur ... ed+logging

Greetings,

Marco

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Sun Sep 11, 2016 6:33 pm

Hallo Marco,

thank you again for your input. Indeed, after adding the ? the example REST Webservice was working. I missed this.

Nevertheless my Neo4j Rest Call is not working. I looked a bit closer and tested the different response types. I am getting an 401 with the following message:

Code: Select all

{
  "errors" : [ {
    "code" : "Neo.ClientError.Security.Unauthorized",
    "message" : "No authentication header supplied."
  } ]
}
This means, I can not use the Arguments params to provide these information.

Just as reminder: I want to pass something like this:

Code: Select all

POST http://localhost:7474/db/data/node/1/paths HTTP/1.1
Authorization: Basic bmVvNGo6bmVvcGFzcw==
Host: localhost:7474

{
  "to" : "http://localhost:7474/db/data/node/5",
  "cost_property" : "cost",
  "relationships" : {
    "type" : "connected_by",
    "direction" : "out"
  },
  "algorithm" : "dijkstra"
}
i. e. the Method "Post", an URL, a header attribute for authorization and a body with the corresponding "instructions".

But I can not see the way, how I should pass the header attributes (e. g. Authorization) and the body with the JSON document.

I enabled the extended logging and set it to TRACE. Thanks for that. But there is not the full REST call, the values printed are just the passed values of the csv-file (e. g. ...node/5).

Do you have some suggestions, especially for my first problem? Without these I am not able to use Neo4j for my prototype.
Is the Rapid Prototyping Feature maybe an option? I tried the Wiki example Java Code with the already known csv file:

Code: Select all

output = JAVA({path='C:\WorkSpace\Odysseus\Testing\Javatest.java'}, src_neo4jTest)
And directly got a Nullpointer Exception.

Greetings,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Mon Sep 12, 2016 1:25 pm

Hi Stefan,

the new nightly build contains an extension of the WSEnrich operator It has a new parameter header that can be used just like the arguments set:

Code: Select all

                    header = [['Authorization', 'Basic bmVvNGo6bmVvcGFzcw==']],


Maybe you can try, if this solves your problem?

The NullPointerException indicates that no java compiler is found. I guess Tools.jar is not awailable in the classpath of the running jvm. I added a better exception message.

Greetings,

Marco

P.S.: There is another update site http://odysseus.offis.uni-oldenburg.de/update2/ where every master code change is pushed to immediatly (not only nightly). This site is not available while local compilation is running.

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Mon Sep 12, 2016 6:38 pm

Hi Marco,

this is great. I tested it and it is working as expected. :) However I am still not able to attach a body (that includes a JSON document) to the REST Call. Furthermore the mandatory Arguments parameter is not needed in this case which means I have to attach it and hopefully it is ignored all the time.

But thank you very much for this modification.

According to Tools.jar: This could be possible. Usually the jre and the jdk are installed together because the jdk should be used for development and the jre as "public vm". In Eclipse you can change the used vm in the preferences. In Odysseus Studio I cannot find a way to change that. The JAVA_PATH environment variable as well as the PATH variable (windows system) already pointing to the java jdk folder and the corresponding bin folder respectively. I copied the tools.jar to the /lib/ext folder in the jre directory, but that did not work.

I know, that are a lot of questions and new/different use cases. I just try to find out how I can achieve these requirements with Odysseus. Thank you very much for your support!

Greetings,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Tue Sep 13, 2016 10:12 am

Hi Stefan,

have you tried: method = 'POST_DOCUMENT'?

Regarding the tools.jar. Changing the odysseus.ini (this is the same as the eclipse.ini) in the start folder of odyseus studio should help

https://wiki.eclipse.org/Eclipse.ini

Code: Select all

-vm
<PATH TO JAVA>
Greetings,

Marco

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest