Bulk Loading and Processing with Coherence

Added by Patrick Peralta, last edited by Gene Gleyzer on Apr 09, 2008  (view change)


Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.

Bulk Writing to a Cache

A common scenario when using Coherence is to pre populate a cache before the application makes use of the cache. A simple way to do this would be:

This works, but each call to put may result in network traffic, especially for partitioned and replicated caches. Additionally, each call to put will return the object it just replaced in the cache (per the java.util.Map interface) which will add more unnecessary overhead. This can be made much more efficient by using putAll instead:

Efficient processing of filter results

Coherence provides the ability to query caches based on criteria via the filter API. Here is an example (given entries with integers as keys and strings as values):

This example works for small data sets, but it may encounter problems if the data set is too large, such as running out of heap space. Here is a pattern to process query results in batches to avoid this problem:

In this example, all keys for entries that match the filter are returned, but only BUFFER_SIZE (in this case, 100) entries are retrieved from the cache at a time.

Note that ${xhtml} can be used to process results in parts, similar to the example above. However LimitFilter is meant for scenarios where the results will be paged, such as in a user interface. It is not an efficient means to process all data in a query result.

A Complete Example

Here is an example program that demonstrates the concepts described above. Note this can be downloaded from the forums.

Here are the steps to running the example:

  1. Save this file as com/tangosol/examples/PagedQuery.java
  2. Point the classpath to the Coherence libraries and the current directory
  3. Compile and run the example

$ export COHERENCE_HOME=[**Coherence install directory**]

$ export CLASSPATH=$COHERENCE_HOME/lib/tangosol.jar:$COHERENCE_HOME/lib/coherence.jar:.

$ javac com/tangosol/examples/PagedQuery.java

$ java com.tangosol.examples.PagedQuery

2007-05-23 12:19:44.156 Oracle Coherence 3.3/387  (thread=main, member=n/a): Loaded operational configuration from
 resource "jar:file:/C:/coherence/lib/coherence.jar!/tangosol-coherence.xml"
2007-05-23 12:19:44.171 Oracle Coherence 3.3/387  (thread=main, member=n/a): Loaded operational overrides from
resource "jar:file:/C:/coherence/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
2007-05-23 12:19:44.171 Oracle Coherence 3.3/387  (thread=main, member=n/a): Optional configuration override
"/tangosol-coherence-override.xml" is not specified

Oracle Coherence Version 3.3/387
 Grid Edition: Development mode
Copyright (c) 2000-2007 Oracle. All rights reserved.

2007-05-23 12:19:44.812 Oracle Coherence GE 3.3/387  (thread=Cluster, member=n/a): Service Cluster joined the cluster
with senior service member n/a
2007-05-23 12:19:48.062 Oracle Coherence GE 3.3/387  (thread=Cluster, member=n/a): Created a new cluster with
Member(Id=1, Timestamp=2007-05-23 12:19:44.609, Address=, MachineId=26828, Edition=Grid Edition,
Mode=Development, CpuCount=2, SocketCount=1) UID=0xC0A800CC00000112B9BC9B6168CC1F98
Adding 1024 entries to cache
Adding 1024 entries to cache

...repeated many times...

Adding 1024 entries to cache
Adding 1024 entries to cache
Adding 1024 entries to cache
processing chunk of 100 contacts:
  [25827]=Contact{Name=Cgkyleass Kmknztk, Phone=(285) 452-0000}
  [4847]=Contact{Name=Cyedlujlc Ruexrtgla, Phone=(255) 296-0000}
...repeated many times
  [33516]=Contact{Name=Cjfwlxa Wsfhrj, Phone=(683) 968-0000}
  [71832]=Contact{Name=Clfsyk Dwncpr, Phone=(551) 957-0000}
processing chunk of 100 contacts:
  [38789]=Contact{Name=Cezmcxaokf Kwztt, Phone=(725) 575-0000}
  [87654]=Contact{Name=Cuxcwtkl Tqxmw, Phone=(244) 521-0000}
...repeated many times
  [96164]=Contact{Name=Cfpmbvq Qaxty, Phone=(596) 381-0000}
  [29502]=Contact{Name=Cofcdfgzp Nczpdg, Phone=(563) 983-0000}
processing chunk of 80 contacts:
  [49179]=Contact{Name=Czbjokh Nrinuphmsv, Phone=(140) 353-0000}
  [84463]=Contact{Name=Cyidbd Rnria, Phone=(571) 681-0000}
  [2530]=Contact{Name=Ciazkpbos Awndvrvcd, Phone=(676) 700-0000}
  [9371]=Contact{Name=Cpqo Rmdw, Phone=(977) 729-0000}