org.apache.flink.contrib.siddhi.operator

Class AbstractSiddhiOperator<IN,OUT>

java.lang.Object

org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

org.apache.flink.contrib.siddhi.operator.AbstractSiddhiOperator<IN,OUT>

Type Parameters:

IN - Input Element Type

OUT - Output Element Type

All Implemented Interfaces:

Serializable, org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.StreamCheckpointedOperator, org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Direct Known Subclasses:

SiddhiStreamOperator

public abstract class AbstractSiddhiOperator<IN,OUT>
extends org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>
implements org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>, org.apache.flink.streaming.api.operators.StreamCheckpointedOperator

Siddhi Runtime Operator

A flink Stream Operator to integrate with native siddhi execution runtime, extension and type schema mechanism/

• Create Siddhi ExecutionPlanRuntime according predefined execution plan and integrate with Flink Stream Operator lifecycle.
• Connect Flink DataStreams with predefined Siddhi Stream according to unique streamId
• Convert native StreamRecord to Siddhi Event according to StreamSchema, and send to Siddhi Runtime.
• Listen output callback event and convert as expected output type according to output TypeInformation, then output as typed DataStream.
• Integrate siddhi runtime state management with Flink state (See `AbstractSiddhiOperator`)
• Support siddhi plugin management to extend CEP functions. (See `SiddhiCEP#registerExtension`)

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

org.apache.flink.streaming.api.operators.AbstractStreamOperator.CountingOutput

Field Summary

Fields inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

chainingStrategy, LOG, metrics, output

Constructor Summary

Constructors

Constructor and Description
AbstractSiddhiOperator(SiddhiOperatorContext siddhiPlan)

Method Summary

Modifier and Type	Method and Description
protected abstract org.apache.flink.streaming.runtime.streamrecord.MultiplexingStreamRecordSerializer<IN>	createStreamRecordSerializer(StreamSchema streamSchema, org.apache.flink.api.common.ExecutionConfig executionConfig)
void	dispose()
PriorityQueue<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN>>	getPriorityQueue()
org.wso2.siddhi.core.stream.input.InputHandler	getSiddhiInputHandler(String streamId)
protected SiddhiOperatorContext	getSiddhiPlan()
protected org.wso2.siddhi.core.ExecutionPlanRuntime	getSiddhiRuntime()
abstract String	getStreamId(IN record)
protected org.apache.flink.streaming.runtime.streamrecord.MultiplexingStreamRecordSerializer<IN>	getStreamRecordSerializer(String streamId)
void	open()
void	processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
protected abstract void	processEvent(String streamId, StreamSchema<IN> schema, IN value, long timestamp)
void	processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
protected abstract PriorityQueue<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN>>	restoreQueuerState(org.apache.flink.core.memory.DataInputView dataInputView)
void	restoreState(org.apache.flink.core.fs.FSDataInputStream state)
protected void	send(String streamId, Object[] data, long timestamp) Send input data to siddhi runtime
void	setup(org.apache.flink.streaming.runtime.tasks.StreamTask<?,?> containingTask, org.apache.flink.streaming.api.graph.StreamConfig config, org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<OUT>> output)
protected abstract void	snapshotQueuerState(PriorityQueue<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN>> queue, org.apache.flink.core.memory.DataOutputView dataOutputView)
void	snapshotState(org.apache.flink.core.fs.FSDataOutputStream out, long checkpointId, long timestamp)
RunnableFuture<org.apache.flink.runtime.state.OperatorStateHandle>	snapshotState(long checkpointId, long timestamp, org.apache.flink.runtime.state.CheckpointStreamFactory streamFactory)

Methods inherited from class org.apache.flink.streaming.api.operators.AbstractStreamOperator

close, getChainingStrategy, getContainingTask, getExecutionConfig, getKeyedStateBackend, getMetricGroup, getOperatorConfig, getOperatorStateBackend, getPartitionedState, getPartitionedState, getRuntimeContext, getTimerService, getUserCodeClassloader, notifyOfCompletedCheckpoint, restoreState, setChainingStrategy, setKeyContext, setKeyContextElement1, setKeyContextElement2

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.flink.streaming.api.operators.StreamOperator

close, getChainingStrategy, getMetricGroup, notifyOfCompletedCheckpoint, restoreState, setChainingStrategy, setKeyContextElement1, setKeyContextElement2

Constructor Detail

AbstractSiddhiOperator

public AbstractSiddhiOperator(SiddhiOperatorContext siddhiPlan)

Parameters:

siddhiPlan - Siddhi CEP Execution Plan

Method Detail

createStreamRecordSerializer

protected abstract org.apache.flink.streaming.runtime.streamrecord.MultiplexingStreamRecordSerializer<IN> createStreamRecordSerializer(StreamSchema streamSchema,
                                                                                                                                       org.apache.flink.api.common.ExecutionConfig executionConfig)

getStreamRecordSerializer

protected org.apache.flink.streaming.runtime.streamrecord.MultiplexingStreamRecordSerializer<IN> getStreamRecordSerializer(String streamId)

processElement

public void processElement(org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN> element)
                    throws Exception

Specified by:

processElement in interface org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>

Throws:

Exception

processEvent

protected abstract void processEvent(String streamId,
                                     StreamSchema<IN> schema,
                                     IN value,
                                     long timestamp)
                              throws Exception

Throws:

Exception

processWatermark

public void processWatermark(org.apache.flink.streaming.api.watermark.Watermark mark)
                      throws Exception

Specified by:

processWatermark in interface org.apache.flink.streaming.api.operators.OneInputStreamOperator<IN,OUT>

Throws:

Exception

getStreamId

public abstract String getStreamId(IN record)

getPriorityQueue

public PriorityQueue<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN>> getPriorityQueue()

getSiddhiRuntime

protected org.wso2.siddhi.core.ExecutionPlanRuntime getSiddhiRuntime()

getSiddhiInputHandler

public org.wso2.siddhi.core.stream.input.InputHandler getSiddhiInputHandler(String streamId)

getSiddhiPlan

protected SiddhiOperatorContext getSiddhiPlan()

setup

public void setup(org.apache.flink.streaming.runtime.tasks.StreamTask<?,?> containingTask,
                  org.apache.flink.streaming.api.graph.StreamConfig config,
                  org.apache.flink.streaming.api.operators.Output<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<OUT>> output)

Specified by:

setup in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

setup in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

open

public void open()
          throws Exception

Specified by:

open in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

open in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

send

protected void send(String streamId,
                    Object[] data,
                    long timestamp)
             throws InterruptedException

Send input data to siddhi runtime

Throws:

InterruptedException

dispose

public void dispose()
             throws Exception

Specified by:

dispose in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

dispose in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

snapshotState

public void snapshotState(org.apache.flink.core.fs.FSDataOutputStream out,
                          long checkpointId,
                          long timestamp)
                   throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamCheckpointedOperator

Throws:

Exception

snapshotState

public RunnableFuture<org.apache.flink.runtime.state.OperatorStateHandle> snapshotState(long checkpointId,
                                                                                        long timestamp,
                                                                                        org.apache.flink.runtime.state.CheckpointStreamFactory streamFactory)
                                                                                 throws Exception

Specified by:

snapshotState in interface org.apache.flink.streaming.api.operators.StreamOperator<OUT>

Overrides:

snapshotState in class org.apache.flink.streaming.api.operators.AbstractStreamOperator<OUT>

Throws:

Exception

restoreState

public void restoreState(org.apache.flink.core.fs.FSDataInputStream state)
                  throws Exception

Specified by:

restoreState in interface org.apache.flink.streaming.api.operators.StreamCheckpointedOperator

Throws:

Exception

snapshotQueuerState

protected abstract void snapshotQueuerState(PriorityQueue<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN>> queue,
                                            org.apache.flink.core.memory.DataOutputView dataOutputView)
                                     throws IOException

Throws:

IOException

restoreQueuerState

protected abstract PriorityQueue<org.apache.flink.streaming.runtime.streamrecord.StreamRecord<IN>> restoreQueuerState(org.apache.flink.core.memory.DataInputView dataInputView)
                                                                                                               throws IOException

Throws:

IOException