Odysseus

This is the official support forum for Odysseus

Working with payload based end timestamps

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Working with payload based end timestamps

Postby stefan » Fri Dec 16, 2016 4:43 pm

Hi,

as the most of you know, I am working on a prototype of a mobility platform. Mostly I use mobility requests and mobility offers. At the moment I can read mobility requests from a csv (simulation of the user interface) and get the required data to answer these requests through processing in Odysseus. Furthermore I can read mobility offers from a csv file and store this information.

Now I want to do the following: Instead of answering a mobility request once some requests (with a special attribute value) should be examined for a period of time (like an search agent). A mobility request could be valid for a specific time period. The start and the end timestamp could be stored in the payload of a tuple. Alternatively one could store how long the request is valid (starttimestamp + delta = endtimestamp). If a new mobility offer comes in, the currently valid mobility requests should be processed again to check if there are new relevant offers for these requests.

I have some ideas to do that, but the first idea was to set the end timestamp based on a payload attribute. I tried to use the ENDTIMESTAMP during creating the source or the timestamp operator to add the end timestamp based on a timestamp in the payload. I always get the error message:

Code: Select all

'Parsing and Executing Query' has encountered a problem.

Script Execution Error: transformation failed ; You cannot use start and end parameters for system time processing


What is the reason for that?

Alternative ideas would be to use the context store or a database for storing mobility requests that should be checked again later on.

My question is mostly about what is the preferred way to do that in Odysseus and why I cannot set the endtimestamp. I understand that the csv source does not provide a starttimestamp and therefore the system time is used for processing, but why it is not possible to use (a different) system time as end timestamp?

thanks & greetings,
Stefan

tbrandt
Posts: 4
Joined: Fri Dec 16, 2016 5:55 pm

Re: Working with payload based end timestamps

Postby tbrandt » Fri Dec 16, 2016 5:59 pm

Hi,

I only have a minute, therefore just a very short answer. Maybe I or someone else can answer in more detail tomorrow or so.

I think what you are searching for are windows. Use the window operator to set the end timestamp. If the source sets the start-timestamp with the system time (e.g. timestamp "10") and you have a time-window of size 10 ms (or a greater value if you use minutes or so) then the end-timestamp will be 20. In short: Use a Window operator. :) More info: http://wiki.odysseus.offis.uni-oldenburg.de/display/ODYSSEUS/Window+operator

Greetings,
Tobias

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Working with payload based end timestamps

Postby stefan » Fri Dec 16, 2016 7:53 pm

Hi Tobias,

thank you for your response!
Yes, I know window operators and actually this should be exactelly what I am looking for. But I am not sure how this can be done. The reason is, I dont want to set the endtimestamp to a fix delta value, e. g. starttimestamp + x ms. I am looking for something that can be defined by the user in the payload.

Imagine a request with id, departure, destination, validUntil (except the id everything should be defined by the user). For the user this fulfills the functionality: "Inform me about each fitting offer until my defined 'latest response' is reached".

I want to set the starttimestamp to the system time when the tuple arrives. The endtimestamp should be equal to validUntil. validUntil can be a fixed timestamp (1481910693) or a delta (+3 h) - depending what is easier to implement or possible. Now I could do a join between the "windowed" requests and the incoming offers. As long as the request would be valid, the join would deliver the request+offer tuple. Each time I get such a tuple I can do my processing the respective request+offer pair.

greetings,
Stefan

cornelius
Posts: 2
Joined: Mon Aug 25, 2014 4:51 pm

Re: Working with payload based end timestamps

Postby cornelius » Fri Dec 16, 2016 8:48 pm

Did you try the timestamp operator? https://wiki.odysseus.informatik.uni-oldenburg.de/display/ODYSSEUS/Timestamp+operator

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Working with payload based end timestamps

Postby stefan » Mon Dec 19, 2016 1:20 pm

Thank you for your response! Yes, I got an error message. See my first post for details and the error message.

User avatar
Marco Grawunder
Posts: 256
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with payload based end timestamps

Postby Marco Grawunder » Mon Dec 19, 2016 2:28 pm

Yes ... ;-)

You can use a "hack" to allow this. Simply, add 'SYSTEMTIME = false' (even if system time is used here). The calculation of the new time must be done before in a MAP operator. You can access the current start time stamp as in the given example

Code: Select all

map = MAP({EXPRESSIONS = [['TimeInterval.Start+1000','EndTS']], KEEPINPUT = true}, nexmark:person)

out = TIMESTAMP({end='EndTS', SYSTEMTIME = false}, map)

cornelius
Posts: 2
Joined: Mon Aug 25, 2014 4:51 pm

Re: Working with payload based end timestamps

Postby cornelius » Mon Dec 19, 2016 2:37 pm

You can also set the start timestamp explicitly in the TIMESTAMP operator to the value that was set by the ACCESS operator:

Code: Select all

#PARSER PQL
#QUERY
input = ACCESS({
                  source='input',
                  wrapper='GenericPull',
                  transport='File',
                  protocol='CSV',
                  datahandler='Tuple',
                  options=[
                    ['delimiter',','],
                    ['textDelimiter',"'"],
                    ['readfirstline','false'],
                    ['delay','100'],
                    ['filename', 'input.csv']
                  ],
                  schema=[
                    ['x','String'],
                    ['delta','Long']
                  ]                                               
                }                                           
              )

withEndTs = MAP({expressions = [ 'x', ['TimeInterval.start', 'startts'], ['TimeInterval.start + delta', 'endts'] ] }, input)
withTimeInterval = TIMESTAMP({start='startts', end='endts'}, withEndTs)

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Working with payload based end timestamps

Postby stefan » Mon Dec 19, 2016 5:05 pm

Hi,

thanks, that should be exactly what I am looking for.
I calculate the value in a MAP operator and set it with the timestamp operator:

Code: Select all

   ...
   ['toLong(TimeInterval.Start + latestResponseDelta)','endTs']
   ...
   
   mrNonbindingTs = TIMESTAMP({end='endTs', SYSTEMTIME = false}, mrNonbinding)


The value is created and looks fine but the endtimestamp remains <null> if I look at it in the stream table. I just get this output in the eclipse console:

Code: Select all

1983395 TRACE DefaultStreamConnection  - Receiving element from port 0: 3cc747e5-05b5-41f0-b2a4-147627ae6b19|MobilityRequestInitial|MR0000000002|1|U0000000101|NonbindingRequest|http://.../Bahnhof|http://.../ZOB|24.10.2016 17:05:00 (GMT+2)|n/a|1477321500|-1|20161024|monday|1705|-1|n/a|-1|1|[]|<NULL>|<NULL>|<NULL>|<NULL>|<NULL>|<NULL>|1800000|1482160687450 | META | 1482158887450|oo - de.uniol.inf.is.odysseus.core.streamconnection.DefaultStreamConnection.notifyListeners(DefaultStreamConnection.java:370)


Am I doing something wrong? I tried it also just by adding 1000 to the start timestamp etc. There is no difference.

greetings,
Stefan

User avatar
Marco Grawunder
Posts: 256
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: Working with payload based end timestamps

Postby Marco Grawunder » Mon Dec 19, 2016 5:43 pm

Oops. I have not looked exactly at the output, sorry. The TimeStamp operator was not able to set only the end time stamp but required always the start time stamp, too.

Is fixed now.

stefan
Posts: 85
Joined: Tue Jul 12, 2016 1:03 pm

Re: Working with payload based end timestamps

Postby stefan » Mon Dec 19, 2016 6:55 pm

Ok, now it is working. Thank you very much!


phpbb 3.1 style demo

Return to “PQL”

Who is online

Users browsing this forum: No registered users and 1 guest