select certain input records in stream

kauerbach
Posts: 30
Joined: Fri Aug 22, 2014 4:52 pm

select certain input records in stream

Post by kauerbach » Mon Sep 22, 2014 10:37 pm

Hello,

I have a data stream that has records of different types, with different 'tuples' for each type. Each record starts with a character 'key' that gives the record type. I would like to only use one type of record as input to Odysseus - that is, only the records beginning with the character 'P'. Those records will all have the same elements. Is there a way to tell Odysseus to only allow those records from the input stream?

Thank you.
Ken Auerbach.

User avatar
ckuka
Posts: 35
Joined: Tue Jul 29, 2014 10:55 am
Location: Shanghai, P.R. China/中国上海
Contact:

Re: select certain input records in stream

Post by ckuka » Tue Sep 23, 2014 4:31 am

Hi,
In your case, the problem is that the data stream consists of CSV strings with different number of elements. However, the current protocol handler for CSV expects a CSV string with a fixed number elements. This behavior should be fixed now by appending the missing values with NULL values. This fix will be included in the next build.

Now to the filtering question, to filter a stream for a given string, you can use the SELECT operator to compare the value of an attribute with another value, e.g.,

Code: Select all

output = SELECT({PREDICATE = 'key == "P"'}, input)
The output stream of the operator will only consists of tuples with a "key" attribute value of "P" (string compare).
kind regards / 祝好
Christian Kuka

kauerbach
Posts: 30
Joined: Fri Aug 22, 2014 4:52 pm

Re: select certain input records in stream

Post by kauerbach » Tue Sep 23, 2014 5:12 pm

Thank you, Christian.
Can this line "output = SELECT({PREDICATE = 'key == "P"'}, input)" be added to my current script or do I need to create another script with this line in it?
Thank you.
Regards,
Ken.

User avatar
ckuka
Posts: 35
Joined: Tue Jul 29, 2014 10:55 am
Location: Shanghai, P.R. China/中国上海
Contact:

Re: select certain input records in stream

Post by ckuka » Tue Sep 23, 2014 5:36 pm

Sure you can add that line to your current script. Of course you have to change the name of the attributes to match the schema of your data stream and change the name of the input to the name you used for the source.
kind regards / 祝好
Christian Kuka

kauerbach
Posts: 30
Joined: Fri Aug 22, 2014 4:52 pm

Re: select certain input records in stream

Post by kauerbach » Wed Sep 24, 2014 7:19 pm

Thank you, Christian. I added that line (with my input defined as "Source"), but I get an error message that it is expecting an operator instead of 'Source'.
My query is below.
Thank you.
Kind regards,
Ken.

--------
#PARSER PQL
#QUERY
input := ACCESS({
source='Source',
wrapper='GenericPush',
transport='NonBlockingTcp',
protocol='CSV',
dataHandler='Tuple',
options=[
['delimiter','\t'],
['readfirstline','false'],
['host','arvi.mgh.harvard.edu'],
['port','1194'],
['logininfo','DOMAINID=15\\nTOPIC=Numeric\\nSTART\\n']],
schema=[
['keyStr','String'],
['topic','String'],
['metricStr','String'],
['instance','String'],
['value','Float'],
['value2','String']]
})
output = SELECT({PREDICATE = 'keyStr == "P"'}, Source)

User avatar
ckuka
Posts: 35
Joined: Tue Jul 29, 2014 10:55 am
Location: Shanghai, P.R. China/中国上海
Contact:

Re: select certain input records in stream

Post by ckuka » Thu Sep 25, 2014 1:39 am

That should do the job:

Code: Select all

output = SELECT({PREDICATE = 'keyStr == "P"'}, input)
More on the PQL syntax can be found at http://odysseus.offis.uni-oldenburg.de: ... Id=4587829
kind regards / 祝好
Christian Kuka

kauerbach
Posts: 30
Joined: Fri Aug 22, 2014 4:52 pm

Re: select certain input records in stream

Post by kauerbach » Fri Sep 26, 2014 5:05 pm

Hi Christian,

I keep getting syntax errors when I include the line "output = SELECT({predicate='keyStr == "P"'}, Source)" in my script. It's expecting an 'Operator' where I have "Source". Could you please send me some example scripts using the "output = SELECT(..." line so that I can construct it properly for my script?

Thank you.
Kind regards,
Ken.

flp
Posts: 51
Joined: Tue Sep 02, 2014 12:54 am

Re: select certain input records in stream

Post by flp » Fri Sep 26, 2014 9:54 pm

You defined your source as input. Therefore, you have to use input instead of Source and it will work.

User avatar
Marco Grawunder
Posts: 272
Joined: Tue Jul 29, 2014 10:29 am
Location: Oldenburg, Germany
Contact:

Re: select certain input records in stream

Post by Marco Grawunder » Mon Sep 29, 2014 8:41 am

To make it clear. 'input' is the string to use. The example from Christian should would directly.

kauerbach
Posts: 30
Joined: Fri Aug 22, 2014 4:52 pm

Re: select certain input records in stream

Post by kauerbach » Wed Oct 01, 2014 6:44 pm

Hello,

I added a seventh attribute to the query and now it's working. I can see the data stream in the Studio window and I can chart the numerical attribute.

Thank you.
Kind regards,
Ken.

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest