Package org.aksw.commons.io.cache

Class RangeRequestWorkerImpl<A>

java.lang.Object
org.aksw.commons.util.closeable.AutoCloseableBase
org.aksw.commons.util.closeable.AutoCloseableWithLeakDetectionBase
org.aksw.commons.io.cache.RangeRequestWorkerImpl<A>
Type Parameters:
A -
All Implemented Interfaces:
AutoCloseable, Runnable
public class RangeRequestWorkerImpl<A> extends AutoCloseableWithLeakDetectionBase implements Runnable
A producer task: Takes items from an iterator and writes them them to pages
Author:
raven

  • Field Details

    • cacheSystem

      protected AdvancedRangeCacheImpl<A> cacheSystem
      Reference to the manager - if there is a failure in processing the request the executor notifies it

    • arrayOps

      protected ArrayOps<A> arrayOps

    • endpointDemands

      protected ObservableSlottedValue<Long,Long> endpointDemands
      Demands from clients to load data up to the supplied value. A client can unregister a demand any time.

    • dataStream

      protected ReadableChannel<A> dataStream
      The data supplying iterator. If all data in the range of operation of this worker is cached then no backend request is made.

    • slice

      protected Slice<A> slice
      The data slice

    • pageRange

      protected SliceAccessor<A> pageRange
      The pages claimed by this worker; obtained from slice

    • currentPageId

      protected long currentPageId

    • requestOffset

      protected final long requestOffset
      The offset of the original request

    • requestLimit

      protected final long requestLimit
      The requestLimit must take result-set-limit on the backend into account!

    • offsetToMaxAllowedRefetchCount

      protected LongUnaryOperator offsetToMaxAllowedRefetchCount
      Metadata supplier to support scheduling: For a given offset return the maximum length of a run of items that have already been retrieved before a separate request should be fired instead of retrieving those items twice and wasting data volume

    • bulkSize

      protected int bulkSize
      The number of items to process in one batch (before checking for conditions such as interrupts or no-more-demand)

    • offset

      protected volatile long offset
      The current offset

    • nextCheckpointOffset

      protected long nextCheckpointOffset
      The offset at which the next checkpoint is sheduled

    • terminationDelay

      protected Duration terminationDelay
      Task termination may be delayed in millis in order to allow it to recover should another observer register in the delay phase

    • idleMode

      protected IdleMode idleMode
      Wait mode - true: do not fetch more data then there is demand - false: keep pre-fetching data

    • terminationTimer

      protected transient com.google.common.base.Stopwatch terminationTimer
      A timer is started as soon as there is no more explicit demand for data. Depending on the idle mode the worker can either pause or read ahead until the termination delay is reached.

    • firstItemTime

      protected Duration firstItemTime
      Time it took to retrieve the first item

    • numItemsProcessed

      protected long numItemsProcessed
      Throughput in items / second

    • processingTimeInNanos

      protected long processingTimeInNanos
      Total time

  • Constructor Details

    • RangeRequestWorkerImpl

      public RangeRequestWorkerImpl(AdvancedRangeCacheImpl<A> cacheSystem, long requestOffset, long requestLimit, int bulkSize, Duration terminationDelay)

  • Method Details

    • getMaxAllowedRefetchCount

      public long getMaxAllowedRefetchCount(long offset)

    • getFirstItemTime

      public Duration getFirstItemTime()
      Time in seconds it took to obtain the first item

    • getThroughput

      public float getThroughput()
      Throughput measured in items per second

    • etaAtIndex

      public float etaAtIndex(long index)
      Estimated time of arrival at the given index in seconds Index must be greater or equal to offset Call #pause() before calling the method.

    • closeActual

      protected void closeActual()
      Stops processing of this executor and also disposes the underlying data supplier (which is expected to terminate)
      Overrides:
      closeActual in class AutoCloseableBase

    • initBackendRequest

      protected void initBackendRequest() throws IOException
      Throws:
      IOException

    • run

      public void run()
      Specified by:
      run in interface Runnable

    • checkpoint

      protected void checkpoint()

    • runCore

      public void runCore() throws Exception
      A worker can only shutdown if it holds the workerSetLock: A concurrent RangeRequestIterator may schedule a new task for a worker that was just about to terminate. Shutdown conditions: - idle for too long - reached end of data - - exception
      Throws:
      Exception

    • getCurrentOffset

      public long getCurrentOffset()

    • getEndOffset

      public long getEndOffset()
      The offset of the first item that won't be served by this worker

    • getWorkingRange

      public com.google.common.collect.Range<Long> getWorkingRange()

    • process

      public int process(A buffer, int bufferOffset, int n) throws Exception
      Process up to n more items.
      Throws:
      Exception

    • newDemandSlot

      public Slot<Long> newDemandSlot()
      Acquire a new slot into which the end-offset of a demanded data range can be put. FIXME The worker may just have terminated; so we need synchronization with the workerSyncLock such that during scheduling workers don't disappear.
      Returns: