Package org.aksw.commons.io.slice

Class SliceWithPagesSyncToDisk<A>

java.lang.Object

org.aksw.commons.io.slice.SliceBase<A>

org.aksw.commons.io.slice.SliceWithPagesSyncToDisk<A>

Type Parameters:

A - The collection type of what to store in the pages. Usually an an array such as Binding[] or byte[].

All Implemented Interfaces:

Slice<A>, SliceMetaDataBasic, SliceWithPages<A>, org.aksw.commons.io.util.Sync, org.aksw.commons.util.range.PageHelper

public class SliceWithPagesSyncToDisk<A>
extends SliceBase<A>
implements SliceWithPages<A>

A file-based slice implementation. The is one metadata file and one 'content' file for each page. The metadata file holds the information about which ranges of data have been loaded in the content files. Whenever a set of pages is updated, the metadata needs to be updated as well.

Field Summary

Fields

Modifier and Type	Field	Description
protected SliceMetaDataWithPages	baseMetaData
protected org.aksw.commons.txn.impl.PathDiffState	baseMetaDataStatus
protected org.aksw.commons.collection.rangeset.RangeSetDelegateMutable<Long>	baseRanges
protected RangeBufferDelegateMutable<A>	liveChanges
protected int	liveGeneration	Attribute for advertising detection of external changes to the base data.
protected SliceMetaDataWithPages	liveMetaData
protected com.google.common.collect.RangeSet<Long>	liveMetaDataLoadedRangesView	Cached view of the most recent loaded ranges Changing the value of 'this.liveMetaData' also changes this view's delegate
protected org.slf4j.Logger	logger
protected org.aksw.commons.txn.impl.PathState	metaDataIdentity
protected org.aksw.commons.store.object.key.api.ObjectStore	objectStore
protected org.aksw.commons.path.core.Path<String>	objectStoreBasePath
protected org.aksw.commons.cache.async.AsyncClaimingCache<Long,BufferView<A>>	pageCache
protected LongFunction<String>	pageIdToFileName
protected int	pageSize
protected RangeBufferDelegateMutable<A>	syncChanges
protected SliceMetaDataWithPages	syncMetaData
protected org.aksw.commons.util.concurrent.ScheduleOnce	syncScheduler

Fields inherited from class org.aksw.commons.io.slice.SliceBase

arrayOps, hasDataCondition, readWriteLock

Constructor Summary

Constructors

Constructor	Description
SliceWithPagesSyncToDisk(ArrayOps<A> arrayOps, org.aksw.commons.store.object.key.api.ObjectStore objectStore, org.aksw.commons.path.core.Path<String> objectStoreBasePath, int pageSize, Duration syncDelay)

Method Summary

Modifier and Type	Method	Description
org.aksw.commons.util.closeable.Disposable	addEvictionGuard(com.google.common.collect.RangeSet<Long> range)	Protect a set of ranges from eviction.
static SliceMetaDataWithPages	copyWithNewRanges(SliceMetaDataWithPages base, com.google.common.collect.RangeSet<Long> rangeSet)	Create an immutable snapshot of the current state of the slice suitable for syncing to disk.
static <A> SliceWithPagesSyncToDisk<A>	create(ArrayOps<A> arrayOps, Path repoPath, org.aksw.commons.path.core.Path<String> objectStoreBasePath, int pageSize, Duration syncDelay)
static <A> SliceWithPagesSyncToDisk<A>	create(ArrayOps<A> arrayOps, org.aksw.commons.store.object.key.api.ObjectStore objectStore, org.aksw.commons.path.core.Path<String> objectStoreBasePath, int pageSize, Duration syncDelay)
ArrayOps<A>	getArrayOps()
com.google.common.collect.RangeMap<Long,List<Throwable>>	getFailedRanges()
com.google.common.collect.RangeSet<Long>	getGaps(com.google.common.collect.Range<Long> requestRange)
Condition	getHasDataCondition()
com.google.common.collect.RangeSet<Long>	getLoadedRanges()
long	getMaximumKnownSize()
protected SliceMetaDataBasic	getMetaData()
long	getMinimumKnownSize()
org.aksw.commons.util.ref.RefFuture<BufferView<A>>	getPageForPageId(long pageId)
long	getPageSize()
ReadWriteLock	getReadWriteLock()
boolean	hasMetaDataChanged()
void	loadMetaData(int pageSize)
BufferView<A>	loadPage(long pageId)
BufferView<A>	loadPages(boolean isEager, Set<Long> pageIds)	Loading a page requires locking the metadata file - by loading pages in batch we can reduce redundant locking and reduce the risk of in-between external changes
int	lockAndSyncMetaData(org.aksw.commons.store.object.key.api.ObjectStoreConnection conn, int fallbackPageSize)	Returns the metadata generation
protected RangeBuffer<A>	newChangeBuffer()
void	setMaximumKnownSize(long maximumKnownSize)
void	setMinimumKnownSize(long minimumKnownSize)
void	sync()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.aksw.commons.util.range.PageHelper

getIndexInPageForOffset, getPageIdForOffset, getPageOffsetForPageId

Methods inherited from interface org.aksw.commons.io.slice.Slice

computeFromMetaData, isComplete, mutateMetaData, readMetaData

Methods inherited from interface org.aksw.commons.io.slice.SliceMetaDataBasic

getKnownSize, setKnownSize, updateMaximumKnownSize, updateMinimumKnownSize

Methods inherited from interface org.aksw.commons.io.slice.SliceWithPages

newSliceAccessor

Field Details

logger

protected org.slf4j.Logger logger

objectStore

protected org.aksw.commons.store.object.key.api.ObjectStore objectStore

objectStoreBasePath

protected org.aksw.commons.path.core.Path<String> objectStoreBasePath

pageCache

protected org.aksw.commons.cache.async.AsyncClaimingCache<Long,BufferView<A>> pageCache

pageSize

protected int pageSize

baseRanges

protected org.aksw.commons.collection.rangeset.RangeSetDelegateMutable<Long> baseRanges

baseMetaData

protected SliceMetaDataWithPages baseMetaData

baseMetaDataStatus

protected org.aksw.commons.txn.impl.PathDiffState baseMetaDataStatus

liveMetaData

protected SliceMetaDataWithPages liveMetaData

liveMetaDataLoadedRangesView

protected com.google.common.collect.RangeSet<Long> liveMetaDataLoadedRangesView

Cached view of the most recent loaded ranges Changing the value of 'this.liveMetaData' also changes this view's delegate

syncMetaData

protected SliceMetaDataWithPages syncMetaData

metaDataIdentity

protected org.aksw.commons.txn.impl.PathState metaDataIdentity

syncScheduler

protected org.aksw.commons.util.concurrent.ScheduleOnce syncScheduler

liveGeneration

protected int liveGeneration

Attribute for advertising detection of external changes to the base data. Modification demands the global write lock. Whenever a change to the base data is detected (during sync) then the generation is incremented. All clients should then re-schedule their read/writes.

liveChanges

protected RangeBufferDelegateMutable<A> liveChanges

syncChanges

protected RangeBufferDelegateMutable<A> syncChanges

pageIdToFileName

protected LongFunction<String> pageIdToFileName

Constructor Details

SliceWithPagesSyncToDisk

public SliceWithPagesSyncToDisk(ArrayOps<A> arrayOps,
 org.aksw.commons.store.object.key.api.ObjectStore objectStore,
 org.aksw.commons.path.core.Path<String> objectStoreBasePath,
 int pageSize,
 Duration syncDelay)

Method Details

getReadWriteLock

public ReadWriteLock getReadWriteLock()

Specified by:

getReadWriteLock in interface Slice<A>

Overrides:

getReadWriteLock in class SliceBase<A>

setMinimumKnownSize

public void setMinimumKnownSize(long minimumKnownSize)

Specified by:

setMinimumKnownSize in interface SliceMetaDataBasic

Overrides:

setMinimumKnownSize in class SliceBase<A>

setMaximumKnownSize

public void setMaximumKnownSize(long maximumKnownSize)

Specified by:

setMaximumKnownSize in interface SliceMetaDataBasic

Overrides:

setMaximumKnownSize in class SliceBase<A>

getMetaData

protected SliceMetaDataBasic getMetaData()

Specified by:

getMetaData in class SliceBase<A>

loadMetaData

public void loadMetaData(int pageSize)

create

public static <A> SliceWithPagesSyncToDisk<A> create(ArrayOps<A> arrayOps,
 org.aksw.commons.store.object.key.api.ObjectStore objectStore,
 org.aksw.commons.path.core.Path<String> objectStoreBasePath,
 int pageSize,
 Duration syncDelay)

create

public static <A> SliceWithPagesSyncToDisk<A> create(ArrayOps<A> arrayOps,
 Path repoPath,
 org.aksw.commons.path.core.Path<String> objectStoreBasePath,
 int pageSize,
 Duration syncDelay)

newChangeBuffer

protected RangeBuffer<A> newChangeBuffer()

getPageForPageId

public org.aksw.commons.util.ref.RefFuture<BufferView<A>> getPageForPageId(long pageId)

Specified by:

getPageForPageId in interface SliceWithPages<A>

getGaps

public com.google.common.collect.RangeSet<Long> getGaps(com.google.common.collect.Range<Long> requestRange)

Specified by:

getGaps in interface SliceMetaDataBasic

hasMetaDataChanged

public boolean hasMetaDataChanged()

lockAndSyncMetaData

public int lockAndSyncMetaData(org.aksw.commons.store.object.key.api.ObjectStoreConnection conn,
 int fallbackPageSize)

Returns the metadata generation

loadPage

public BufferView<A> loadPage(long pageId)

loadPages

public BufferView<A> loadPages(boolean isEager,
 Set<Long> pageIds)

Loading a page requires locking the metadata file - by loading pages in batch we can reduce redundant locking and reduce the risk of in-between external changes

getArrayOps

public ArrayOps<A> getArrayOps()

Specified by:

getArrayOps in interface Slice<A>

Overrides:

getArrayOps in class SliceBase<A>

copyWithNewRanges

public static SliceMetaDataWithPages copyWithNewRanges(SliceMetaDataWithPages base,
 com.google.common.collect.RangeSet<Long> rangeSet)

Create an immutable snapshot of the current state of the slice suitable for syncing to disk. This method only requires brief locking in order to set up a new layer of change tracking on top of the existing one.

Throws:

IOException

sync

public void sync()
          throws IOException

Specified by:

sync in interface org.aksw.commons.io.util.Sync

Throws:

IOException

getLoadedRanges

public com.google.common.collect.RangeSet<Long> getLoadedRanges()

Specified by:

getLoadedRanges in interface SliceMetaDataBasic

Overrides:

getLoadedRanges in class SliceBase<A>

getMinimumKnownSize

public long getMinimumKnownSize()

Specified by:

getMinimumKnownSize in interface SliceMetaDataBasic

Overrides:

getMinimumKnownSize in class SliceBase<A>

getMaximumKnownSize

public long getMaximumKnownSize()

Specified by:

getMaximumKnownSize in interface SliceMetaDataBasic

Overrides:

getMaximumKnownSize in class SliceBase<A>

getHasDataCondition

public Condition getHasDataCondition()

Specified by:

getHasDataCondition in interface Slice<A>

Overrides:

getHasDataCondition in class SliceBase<A>

getPageSize

public long getPageSize()

Specified by:

getPageSize in interface org.aksw.commons.util.range.PageHelper

Specified by:

getPageSize in interface SliceWithPages<A>

getFailedRanges

public com.google.common.collect.RangeMap<Long,List<Throwable>> getFailedRanges()

Specified by:

getFailedRanges in interface SliceMetaDataBasic

Overrides:

getFailedRanges in class SliceBase<A>

addEvictionGuard

public org.aksw.commons.util.closeable.Disposable addEvictionGuard(com.google.common.collect.RangeSet<Long> range)

Description copied from interface: Slice

Protect a set of ranges from eviction. If the slice does make use of eviction then this method can return null. Otherwise, a disposable must be returned. As long as it is not disposed no data in the range may get lost due to eviction. This method should not be used directly but via

invalid @link

{@link SliceAccessor#addEvictionGuard(RangeSet))

}.

Specified by:

addEvictionGuard in interface Slice<A>