Communication with Neo4j graph database

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Tue Sep 13, 2016 7:54 pm

Hi Marco,

thank you for your response.
Yes, I tried it and it seems to work except one thing. I am not able to pass the body:

Code: Select all

{
  "to" : "http://localhost:7474/db/data/node/5",
  "cost_property" : "cost",
  "relationships" : {
    "type" : "connected_by",
    "direction" : "out"
  },
  "algorithm" : "dijkstra"
}
I tried to pass it as header attribute as well as argument. I always get an BadInputException. I get this Exception when I don't pass the body. I will have a look if it is possible to log the REST Calls in Neo4j to see whats going on. But I think that the body is simply missing.

According to the tools.jar.
I found the option, but the file is called studio.ini, not odysseus.ini. I just added the -vm + JDK-Path. Now it seems to work but I will check it the next days.

Thank you very much!

Greetings,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database (edited)

Post by Marco Grawunder » Wed Sep 14, 2016 8:57 am

Hi Stefan,

I guess you are right. I added some more logging (trace) output. Maybe you can see what is sent and compare it with what you are expecting. When I look at the code, I think I have an idea to fix this.

Could you please try to add another header: ["Content-Type","application/json"]

and let the parameter contain the json document?

Maybe, a template wourld help,where some values in the json doc are variables that could be replaced by data input?

Greetings,

Marco

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Thu Sep 15, 2016 9:41 am

So, I added the parameter template to the WSEnrich-Operator (available tomorrow)

Code: Select all

neo4jResponse = WSENRICH(   {			
                    servicemethod = 'REST',
                    method = 'POST_DOCUMENT',
                    url = 'http://jsonplaceholder.typicode.com/users',
                    arguments = [['Authorization', 'Authorization']],
                    TEMPLATE = 'Hier ist mal ein toller "test" mit der Authorization',
                    header = [['Authorization', 'Basic bmVvNGo6bmVvcGFzcw==']],
                    parsingmethod = 'JSONPATH',
                    datafields = [
                      ['$.[0].name', 'String']
                    ]                                
                  },
                  src_neo4jTest
                )
Every key the the arguments field is replaced with every occurence in TEMPLATE with the current value of the value field in arguments.
If Authorization contains: Basic bmVvNGo6bmVvcGFzcw==
The body of the request will contain:
Hier ist mal ein toller "test" mit der Basic bmVvNGo6bmVvcGFzcw==

Greetings,

Marco

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Sat Sep 17, 2016 4:29 pm

Hi Marco,

this is amazing! It works as expected. This is my full query:

Code: Select all

#PARSER PQL
#RUNQUERY

neo4jResponse = WSENRICH(
	{
		SERVICEMETHOD = 'REST',
		METHOD = 'POST_DOCUMENT',
		URL = 'http://localhost:7474/db/data/node/1/paths?',
		HEADER = [
			['Authorization', 'Basic bmVvNGo6bmVvcGFzcw=='],
			['Content-Type','application/json']
		],
		TEMPLATE = '{
			  "to" : "to_val",
			  "cost_property" : "cost",
			  "relationships" : {
			    "type" : "connected_by",
			    "direction" : "out"
			  },
			  "algorithm" : "dijkstra"
			}',
		ARGUMENTS = [['to_val', 'to_val']],
		PARSINGMETHOD = 'JSONPATH',
		DEBUG = true,
		DATAFIELDS = [
			['$.[0].nodes[1]', 'String']
		]
	}, 
	src_neo4jTest
)
Now, there is just one small thing. Is it possible to set the URL value via code? The current value "http://localhost:7474/db/data/node/1/paths?" not always the starting point. This means, that I have to change the "1" in the URL depending on the incoming event. I already tried it through the arguments, but that is not working.

The response of this REST Call is a subgraph. I was looking for a way to serialize this as a RDF and attach it to the event. Is this possible?

Thanks so much!
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Mon Sep 19, 2016 11:28 am

Hi Stefan,

i added a new boolean parameter: templateURL. It this is true, the uri will be parsed for replacements in the arguments set.

Code: Select all

                    url = 'http://jsonplaceholder.typicode.com/users/ReplacePath/',
                    TEMPLATEURL = true,
                    arguments = [['Authorization', 'Authorization'],['ReplacePath','path']],
RDF serialization is currently not supported (and not so easy to do). What you can try is the option keyValueOutput=true. If this is set, the results with be a KeyValueObject (more or less the same as JSON, but with another internal repräsentation).

I am not sure, what the result is. Maybe you try and post the result here?

Greetings,

Marco

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Mon Sep 19, 2016 10:44 pm

Hi Marco,

after some trouble with Odysseus (see my other post) I was able to try your changes.
The TEMPLATEURL is working fine, except one small thing: If the URL parameter starts with the argument name that should be replaced, it is not working. If I use it somewhere else in the string, it is working. To clarify:

Not working (from_val = 'http://localhost:7474/db/data/node/1'):

Code: Select all

		URL = 'from_val/paths?',
		ARGUMENTS = [
			['from_val', 'from_val'],
			['to_val', 'to_val']
		],
Working (from_val = 'db/data/node/1'):

Code: Select all

		URL = 'http://localhost:7474/from_val/paths?',
		ARGUMENTS = [
			['from_val', 'from_val'],
			['to_val', 'to_val']
		],
But thats no problem in my case, I just noticed it. :)

My result is a JSON as well:

Request:

Code: Select all

#PARSER PQL
#RUNQUERY

neo4jResponse = WSENRICH(
	{
		SERVICEMETHOD = 'REST',
		METHOD = 'POST_DOCUMENT',
		TEMPLATEURL = true,
		URL = 'http://localhost:7474/from_val/paths?',
		HEADER = [
			['Authorization', 'Basic bmVvNGo6bmVvcGFzcw=='],
			['Content-Type','application/json']
		],
		TEMPLATE = '{
			  "to" : "to_val",
			  "cost_property" : "cost",
			  "relationships" : {
			    "type" : "connected_by",
			    "direction" : "out"
			  },
			  "algorithm" : "dijkstra"
			}',
		ARGUMENTS = [
			['from_val', 'from_val'],
			['to_val', 'to_val']
		],
		PARSINGMETHOD = 'JSONPATH',
		DEBUG = true,
		KEYVALUEOUTPUT = true,
		DATAFIELDS = [
			['$.[0].nodes[1]', 'String'],
			['*', 'String']
			///['$.exception', 'String']
		]
	}, 
	src_neo4jTest
)
Result:

Code: Select all

[
  {
    "relationships": [
      "http://localhost:7474/db/data/relationship/0",
      "http://localhost:7474/db/data/relationship/1"
    ],
    "nodes": [
      "http://localhost:7474/db/data/node/1",
      "http://localhost:7474/db/data/node/4",
      "http://localhost:7474/db/data/node/5"
    ],
    "directions": [
      "->",
      "->"
    ],
    "length": 2,
    "start": "http://localhost:7474/db/data/node/1",
    "weight": 2,
    "end": "http://localhost:7474/db/data/node/5"
  },
  {
    "relationships": [
      "http://localhost:7474/db/data/relationship/2",
      "http://localhost:7474/db/data/relationship/3"
    ],
    "nodes": [
      "http://localhost:7474/db/data/node/1",
      "http://localhost:7474/db/data/node/7",
      "http://localhost:7474/db/data/node/5"
    ],
    "directions": [
      "->",
      "->"
    ],
    "length": 2,
    "start": "http://localhost:7474/db/data/node/1",
    "weight": 2,
    "end": "http://localhost:7474/db/data/node/5"
  }
]
The result will change. That is just some simplified data.

I tried the keyValueOutput=true and keyValueOutput=false. To be honest, I can not see a difference in both settings. I always get the string selected via the JSONPATH statement. Is it possible to work with this JSON stored in one attribute via the KeyValue Feature? Or do you recommend/mean something different?

Thank you for your changes on the WSEnrich operator. I think, now it should be possible to use Neo4j (and other REST Services) easily. Thank you for your help, I really appreciate that!

greetings,
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Tue Sep 20, 2016 4:04 pm

Hi Stefan,

all right. I will create a ticket for the first issues ... I am not sure, how this can happen.

Regarding KeyValue. Indeed there is currently no way to use the KeyValue-Feature together with the WSEnrich. Maybe, a KeyValueObject as part of a tuple would be a solution. We have to think about this.

What do you want to do with the enriched output? Should it be processed with odysseus?

Greetings,

Marco

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Wed Sep 21, 2016 2:13 pm

There are now some new mep functions to handle KeyValueObjects:

Code: Select all

#PARSER PQL
#ADDQUERY
in = TIMER({
          period = 1000,
          source = 'timer'
        }
      )

map = MAP({
          expressions = [
['toKeyValue("{\'title\':\'test\',\'id\':1,\'dateAdded\':1349480292312000,\'lastModified\':1349654494533000,\'type\':\'text/x-moz-place-container\',\'root\':\'placesRoot\'}")','kvObject']
          ]
        },
        in
      )
      
v1 = map({EXPRESSIONS = [['getElement(kvObject,"dateAdded")','dataAdded']]},map)
v2 = map({EXPRESSIONS = [['getElements(kvObject,"title, dateAdded, type")','dataAdded']]},map)

out = CSVFILESINK({SINK = 'sink', FILENAME = '${PROJECTPATH}/out.csv'}, map)

The timer source is just to avoid a data source with json input and trigger the following map operator.

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Communication with Neo4j graph database

Post by stefan » Wed Sep 21, 2016 3:32 pm

Hi Marco,

the respone includes some alternative routes from point A to point B. Based on the response I want to calculate the trip times (duration), the costs (price for the trip) and how often the user has to switch the transportation mode. The "best" alternatives should be sent to the user (the data is processed by an independent receiver. Futhermore I have to create a new event based on the information provided by this JSON reply.

Up to now, I don't know exactly how good I can filter these routes in Neo4j and what processing steps are needed in Odysseus to extract the correct information. The idea is to get the most accurate reply from Neo4j. If this is not possible, I have to recalculate some values (e. g. the cost, time and switch values) in Odysseus and filter some of the routes in Odysseus. This means I have to process the JSON reply.

If it is possible to work with expressions and process this information as described above, this should be fine. I think, now I have to start the development with realistic data and try the map functions for the KeyValueObjects. I would be glad, if I could ask again when I am running into problems.

Thank you very much!
Stefan

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Communication with Neo4j graph database

Post by Marco Grawunder » Wed Sep 21, 2016 4:39 pm

Hi Stefan,

I am very interested in your results and I would be glad to hear about them.

Questions are very welcome, other can learn from them, too and we can improve some functions of Odysseus.

Greetings,

Marco

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest