Working with JSON

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with JSON

Post by Marco Grawunder » Fri Sep 19, 2014 7:58 am

I think the problem could be, that the reading of the elements is too fast. We use timestamps to simulate element based windows, but when den elements have the same timestamps there is no way to distinguish them. In your case, timestamps are based on system time and the granularity is milliseconds.

Could you please add in the source definition the option ['scheduler.delay','1'] and test again?

You shoud now get something like:

Code: Select all

1|10|10|10.0 | META | 1411113161849|1411113161866
2|11|10|21.0 | META | 1411113161866|1411113161881
3|12|10|33.0 | META | 1411113161881|1411113161897
4|13|10|46.0 | META | 1411113161897|1411113161913
5|14|10|60.0 | META | 1411113161913|1411113161928
6|15|10|75.0 | META | 1411113161928|1411113161944
7|16|10|91.0 | META | 1411113161944|1411113161960
8|17|10|108.0 | META | 1411113161960|1411113161975
9|18|10|126.0 | META | 1411113161975|1411113161991
10|19|10|145.0 | META | 1411113161991|1411113162006
10|20|11|155.0 | META | 1411113162006|1411113162007

This is because of the sliding window. For each new element the window is moved one position. So in the first window, there is only the first element, then the first and the second and so on. Finally, the last two elements are from a full window of ten elements.

If you want distinct windows, you can use the SLIDE-Parameter, e.g. SLIDE = 10.

If you only want windows with 10 elements you could e.g. use a selection on the count attribute after the aggregation.

harpreet
Posts: 7
Joined: Tue Sep 16, 2014 6:27 pm

Re: Working with JSON

Post by harpreet » Fri Sep 19, 2014 2:13 pm

Thank You so much. It worked.

I'll trouble you guys again if I get stuck again. Hope you don't mind.


Thanks,
Harpreet

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with JSON

Post by Marco Grawunder » Fri Sep 19, 2014 2:39 pm

Thank You so much. It worked.
That's fine :-)
I'll trouble you guys again if I get stuck again. Hope you don't mind
Hey, thats whats this forum is for ;-)

daniela_4444
Posts: 3
Joined: Sat Mar 11, 2017 12:34 am

Re: Working with JSON

Post by daniela_4444 » Sat Mar 11, 2017 12:41 am

Hi everyone

I am new here and I would like to create a JSON stream source in Odysseus. Unfortunately I always receive the following error:

'Parsing and Executing Query' has encountered a problem.

Script Execution Error: No such data handler: KeyValueObject
No such data handler: KeyValueObject


Does anyone have an idea why I receive this error?

Thank you in advance.

Best regards,
Daniela

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with JSON

Post by Marco Grawunder » Sat Mar 11, 2017 9:20 am

Hi,

how do you use Odysseus? Are you using it as the downloaded product? Then you will need to install the KeyValue Feature

https://wiki.odysseus.informatik.uni-ol ... w+features

When you are using Odysseus from source code, you need to add the feature to the used product and call synchronize.

Greetings,

Marco

daniela_4444
Posts: 3
Joined: Sat Mar 11, 2017 12:34 am

Re: Working with JSON

Post by daniela_4444 » Fri Mar 31, 2017 9:18 am

Hi Marco

thanks a lot, it is now working. Now I tried to use a JSON file as source but unfortunately my PQL/CQL queries do not work. I always receive a null pointer exception. Where do I have to store the JSON file or what may be the problem in my case? I already tried the example from your last posts.

My query looks as follows:
#PARSER PQL
#RUNQUERY
///Your first pql-query here
mysource := ACCESS({
source='sourceName',
wrapper='GenericPull',
transport='File',
protocol='JSON',
dataHandler='KeyValueObject',
options=[['filename','C:/Users/Daniela/workspace/myproject/sample.json']]})

My example JSON file looks as follows:
{"person":[
{"firstName":"a", "age":"27"},
{"firstName":"b", "age":"28"},
{"firstName":"c", "age":"24"},
{"firstName":"d", "age":"25"},
{"firstName":"e", "age":"10"},
{"firstName":"f", "age":"56"},
{"firstName":"g", "age":"45"},
{"firstName":"h", "age":"36"},
{"firstName":"i", "age":"31"},
{"firstName":"j", "age":"35"},
{"firstName":"x", "age":"25"}]}

Thank you in advance!

Best regards,
Daniela

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with JSON

Post by Marco Grawunder » Fri Mar 31, 2017 9:36 am

Hi Daniela,

instead of

options=[['filename','C:/Users/Daniela/workspace/myproject/sample.json']]})

you can use

options=[['filename','${PROJECPATH}/sample.json']]})

What is the null pointer message?

daniela_4444
Posts: 3
Joined: Sat Mar 11, 2017 12:34 am

Re: Working with JSON

Post by daniela_4444 » Sat Apr 01, 2017 12:46 pm

Hi Marco

thank you for your message. Unfortunately it still does not work. I always receive a java.lang.NullPointerException. I also tried to use a csv file as source. It seems to work but I do net see it in my source list. It is always empty. Do you have an idea what could be the problem?

Thank you in advance.

Regards,
Daniela

This is my code for the csv source:
#PARSER PQL
#RUNQUERY
///Your first pql-query here
input = ACCESS({
source='test',
wrapper='GenericPull',
transport='File',
protocol='CSV',
datahandler='Tuple',
options=[
['delimiter',','],
['textDelimiter',"'"],
['readfirstline','true'],
['delay','100'],
['filename', '${PROJECTPATH}/test.csv']
],
schema=[
['firstName','String'],
['age','Integer']
]
}
)

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with JSON

Post by Marco Grawunder » Sat Apr 01, 2017 3:31 pm

Are you using the latest Odysseus version? If not, please update.

Else: Could you please make a bug report?

https://wiki.odysseus.offis.uni-oldenbu ... port+a+bug

Greetings,

Marco

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests