didipostman (2) [Avatar] Offline
#1
As in

http://forum.spring.io/forum/spring-projects/batch/53803-executing-multiple-jobs-at-the-same-time

we can do the same with an oracle paginated query in the item reader side so each spring batch instance will deal with a range of amount of data in input

3. HowTo

• First of all let’s say our SB (Spring batch program is ready) Reader, processor, Writer under c:/gamma4/

• GridGain may be didnt work so get the gridgain-license.xml of GridGain-3.0.9e-win and drop it down into GridGain-3.0.9c-win that what I did last time.

• Set environment variables GridGain_HOME for me it is (c: gridgain-3.0.9c-win)

Here is the code of GridHelloWorldExample.java
And to be run first.

// Copyright (C) GridGain Systems, Inc. Licensed under GPLv3, http://www.gnu.org/licenses/gpl.html

/* _________ _____ __________________ _____
* __ ____/___________(_)______ /__ ____/______ ____(_)_______
* _ / __ __ ___/__ / _ __ / _ / __ _ __ `/__ / __ __ \r
* / /_/ / _ / _ / / /_/ / / /_/ / / /_/ / _ / _ / / /
* \____/ /_/ /_/ \_,__/ \____/ \__,_/ /_/ /_/ /_/
*/

//package org.gridgain.examples.helloworld.api30;

import org.gridgain.grid.*; import org.gridgain.grid.lang.*; import org.gridgain.grid.logger.*;

//import org.gridgain.grid.typedef.X; import org.gridgain.grid.typedef.*; import java.util.*;

import java.util.concurrent.*; import java.io.*;

import static org.gridgain.grid.GridClosureCallMode.*;

/**

* Demonstrates a simple use of GridGain 3.0 APIs.
*


* <h1 class="header">Starting Remote Nodes</h1>
* To try this example you should (but don't have to) start remote grid instances.

* You can start as many as you like by executing the following script:
* <pre class="snippet">{GRIDGAIN_HOME}/bin/ggstart.{bat|sh}</pre>
* Once remote instances are started, you can execute this example from

* Eclipse, IntelliJ IDEA, or NetBeans (and any other Java IDE) by simply hitting run
* button. You will see that all nodes discover each other and
* some of the nodes will participate in task execution (check node
* output).

*


* <h1 class="header">XML Configuration</h1>
* If no specific configuration is provided, GridGain will start with

* all defaults. For information about GridGain default configuration
* refer to {@link GridFactory} documentation. If you would like to
* try out different configurations you should pass a path to Spring
* configuration file as 1st command line argument into this example.

* The path can be relative to {@code GRIDGAIN_HOME} environment variable.
* You should also pass the same configuration file to all other
* grid nodes by executing startup script as follows (you will need

* to change the actual file name):
* <pre class="snippet">{GRIDGAIN_HOME}/bin/ggstart.{bat|sh} examples/config/specific-config-file.xml</pre>
*


* GridGain examples come with multiple configuration files you can try.

* All configuration files are located under {@code GRIDGAIN_HOME/examples/config}
* folder. You are free to try any of these configurations, but whenever
* using 3rd party configurations, such as JBoss JMS, ActiveMQ JMS, Sun MQ JMS, or GigaSpaces,

* make sure to download these respective products and include all the necessary
* libraries into classpath at node startup. All these libraries are already
* specified in commented format in {@code GRIDGAIN_HOME/bin/setenv.{bat|sh}} files
* which get executed automatically by GridGain startup scripts. You can simply

* uncomment the necessary classpath portions as you need.
*
* @author 2005-2011 Copyright (C) GridGain Systems, Inc.

* @version 3.0.9c.19052011
*/

public final class GridHelloWorldExample { /**

* Execute {@code HelloWorld} example on the grid.

*
* @param args Command line arguments, none required but if provided

* first one should point to the Spring XML configuration file. See
* {@code "examples/config/"} for configuration file examples.
* @throws GridException If example execution failed.
*/

static int Cnt =1; static int gridSize = 0;

public static void main(String[] args) throws GridException { if (args.length == 0) {
G.start();
}

else { G.start(args[0]);

}

try {
String ayaUssu = "Wadi benHoucine benAlHabib Mami!" ;
String[] words = ayaUssu.split(" ");

for (String word : words)
{
gridSize++;

}
spreadWordsClosure("1 2 3 4 5");


}
catch(Exception e)
{
System.out.println("exception"+e.toString());

}

finally { G.stop(true);

}
}

/**
* Prints every word a phrase on different nodes.

*
* @param phrase Phrase from which to print words on different nodes.
* @throws GridException If failed.
*/
private static void spreadWordsClosure(String phrase) throws GridException {

// X.println(">>> Starting spreadWordsClosure() example..."smilie;

// Splits the passed in phrase into words and prints every word
// on a individual grid node. If there are more words than nodes -
// some nodes will print more than one word.

G.grid().run(SPREAD, F.yield(phrase.split(" "), new GridInClosure<String>() { @Override public void apply(String word) {

String medm= "C:\jdk1.6.0_17\bin\java -jar D:\steg\RD\GridGainM\src\BATCH_LOT4-2.0.1.RELEASE-jar-with-dependencies.jar job_mensdom.xml "+

"job_mens noparam=xxx date=10/12/2012 lastDateTrait=10/12/2012 userid=53278 useruf=071 "+ " gridsize="+gridSize+" node="+word+" ver="+word+" path=c:/out/";

try {

final Process process = Runtime.getRuntime().exec(medm); X.println(medm);

}

catch(Exception e)
{
System.out.println("exception"+e.toString());

}


X.println(word);
}

}));

// NOTE:

//
// Alternatively, you can use existing closure 'F.println()' to
// print any yield result in 'F.yield()' like so:
//

// G.grid().run(SPREAD, F.yield(phrase.split(" "), F.println()));

//

/* X.println(">>>"smilie;

X.println(">>> Finished printing individual words on different nodes based on GridGain 3.0 API."smilie; X.println(">>> Check all nodes for output (this node is also part of the grid)."smilie;
X.println(">>>"smilie;*/

}

/**

* Ensure singleton. */

private GridHelloWorldExample() { // No-op.

}
}

• Here is the ItemReader code

public class MensItemReader extends SimpleJdbcDaoSupport implements ItemReader<District>, ItemStream {

private static final Log log = LogFactory.getLog(MensItemReader.class);

//DB Fields:

private static final String RC_UR_CODE="RC_UR_CODE"; private static final String libelle_fr="libelle_fr";

private static final String RC_REGION_CODREG="RC_REGION_CODREG"; private static final String LAST_DATE="LAST_DATE";

private static String node="1"; private static int page=0;

private static final String SQL_LIST_District_Count= " select count(*) from rc_ur u where u.rc_ur_code in ( select t.RC_UR_CODE from cf_abonne t )";

private static List<District> lot; private static Iterator<District> it; private static int readIndex=0;

private static String gridsize="1";

//params

private static String userid=null; private static String useruf=null;

private static String date=null; private static Date dateObject=null;

private static String lastDateTrait=null; private static Date lastDateTraitObject=null;

private static Date last_date_traitement=null;

/**

* @see org.springframework.batch.item.ItemReader#read() */

public District read() throws Exception, UnexpectedInputException, ParseException { if (readIndex == lot.size()) {

return null;
}

District d=lot.get(readIndex); d.setUserid(userid); d.setUseruf(useruf); d.setDate_traitement(dateObject);

d.setLast_date_traitement(last_date_traitement); readIndex++;
return d;
}

/**

* @see org.springframework.batch.item.ItemStream#close() */
public void close() throws ItemStreamException {

}

/**

* @see org.springframework.batch.item.ItemStream#open(org.springframework.batch.item.ExecutionContext) */
@SuppressWarnings("unchecked")

public void open(ExecutionContext executionContext) throws ItemStreamException{ log.debug("Method Open");

String SQL_LIST_DISTRICT=

"select * from ( select rownum as rn, u.RC_UR_CODE, u.rc_ur_libelle_fr libelle_fr, u.RC_REGION_CODREG from rc_ur u where
u.rc_ur_code in ( select t.RC_UR_CODE from cf_abonne t ))"+

"where rn >=(("+node+"-1)*"+page+"+1) and rn <=("+node+"*"+page+"smilie";

lot= (List<District>smiliegetJdbcTemplate().query( //SQL QUERY

SQL_LIST_DISTRICT, //ParameterizedRowMapper new RowMapper() {

public District mapRow(ResultSet rs, int rowNum) throws SQLException { if(rs == null){
return null;

}

District district=new District(); district.setRc_ur_code(rs.getString(RC_UR_CODE)); district.setRc_ur_libelle_fr(rs.getString(libelle_fr)); district.setRC_REGION_CODREG(rs.getString(RC_REGION_CODREG)); return district;

}

});

//read date of last traitement

//Date lastTraitDate;

//lastTraitDate=lastDateTraitObject;

//log.info("lastTraitDate is : "+lastTraitDate);

last_date_traitement=lastDateTraitObject; log.info("last_date_traitement: "+last_date_traitement); log.info(SQL_LIST_DISTRICT);

if(lot==null){

lot=new ArrayList<District>();
}

log.info("lot size " + lot.size()); it=lot.iterator();

readIndex=0;
}

/**

* @see org.springframework.batch.item.ItemStream#update(org.springframework.batch.item.ExecutionContext) */

public void update(ExecutionContext executionContext) throws ItemStreamException {

}

public String getNode() { return node;

}

public void setNode(String node) { this.node = node;

//Integer.parseInt(node); int min = 0;

int max = getJdbcTemplate().queryForInt(SQL_LIST_District_Count); this.page = max/Integer.parseInt(this.gridsize);
}

public int getPage() { return page;
}

public String getGridsize(){

return gridsize;

}

public void setGridsize(String gridsize){

this.gridsize = gridsize; //Integer.parseInt(gridsize);

}

public String getUserid() { return userid;
}

public void setUserid(String userid) { this.userid = userid;
}

public String getUseruf() { return useruf;
}

public void setUseruf(String useruf) { this.useruf = useruf;

}

public String getDate() { return date;
}

public void setDate(String date) throws Exception { this.date = date;

//keep this conversion: then, this date can be casted to java.sql.Date dateObject=new java.sql.Date(SyncDateUtil.parse(date,"dd/MM/yyyy").getTime());
}


public String getLastDateTrait() { return lastDateTrait;
}

public void setLastDateTrait(String lastDateTrait)throws Exception { this.lastDateTrait = lastDateTrait;

lastDateTraitObject=new java.sql.Date(SyncDateUtil.parse(lastDateTrait,"dd/MM/yyyy").getTime());
}


//SQL QUERY

private static final String READ_LastTraitDate = "select max(e.bt_eq_dat) "+LAST_DATE+" "+ "from bt_equation e ";

/**
* if abonne isFound_in_rc_etab
* @param district

* @throws Exception
*/

public Date readLastTraitDate() throws Exception{ PreparedStatement stmt = null;

Connection con = null; SQLException exception=null; ResultSet rs;

Date result=null;

try {

// Get Connection object con = getConnection();

stmt = con.prepareStatement(READ_LastTraitDate);

//execute query rs=stmt.executeQuery();

//get the first result if(rs.next()){
result=rs.getDate(LAST_DATE);

}

} catch (SQLException se) { if (log.isErrorEnabled()) {

log.error("SQL Exception: "+ se.toString());
}

exception=se; } finally {

try {

if (stmt != null) { stmt.close();

}

} catch (SQLException se) { if (log.isErrorEnabled()) {
log.error("SQLException while closing objects :
" + se.toString());

}
}
}

if(exception!=null) throw exception; log.info("readLastTraitDate return "+result); return result;

}

}
• Here is piece of job_mensdom.xml

<?xml version="1.0" encoding="UTF-8"?>

<beans:beans xmlns="http://www.springframework.org/schema/batch" xmlns:beans="http://www.springframework.org/schema/beans" xmlns:aop="http://www.springframework.org/schema/aop" >

xmlns:tx="http://www.springframework.org/schema/tx" xmlnssmiliesi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-beans-2.5.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.0.xsd http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.5.xsd">

<beans:import resource="launch-context.xml"/>

<beans:bean id="commandLineJobLauncher" class="com.tn.steg.batch.job.CommandLineJobRunner"> <beans:property name="dataSource" ref="dataSource"/>

</beans:bean>

<beans:bean id="mensdomDAO" class="com.tn.steg.batch.service.MensDomDAO"> <beans:property name="dataSource" ref="dataSource" />

</beans:bean>

<beans:bean id="mensdomManager" class="com.tn.steg.batch.service.impl.MensDomManagerImpl"> <beans:property name="mensdomDao" ref="mensdomDAO" />
</beans:bean>

<beans:bean id="dynamicJobParameters" class="com.tn.steg.batch.job.DynamicJobParameters" />

<beans:bean id="taskExecutor" class="com.tn.steg.batch.partition.SimpleAsyncTaskExecutor" />


<!-- ***************************************************************************************** --> <!-- <br /> <br /> JOBS <br /> <br /> --> <!-- ***************************************************************************************** -->

<!-- params: noparam=xxx useruf={xxx} userid={xxxxx} date={xx/xx/xxxx} lastDateTrait={xx/xx/xxxx, auto}--> <job id="job_mens" job-repository="jobRepository" incrementer="dynamicJobParameters" >

<step id="traiter_Client" >
<tasklet transaction-manager="transactionManager" >

<chunk reader=" ClientItemReader " processor="ClientItemProcessor" writer="ClientWriter" task-executor="taskExecutor" throttle-limit="5" commit-interval="150" >

</chunk>

<transaction-attributes propagation="REQUIRED" isolation="DEFAULT"/> </tasklet>

</step>
</job>

<beans:bean id="ClientItemReader" scope="step" class="com.tn.steg.batch.MensItemReader"> <beans:property name="dataSource" ref="dataSource" />

<beans:property name="userid" value="#{jobParameters['userid']}" /> <beans:property name="useruf" value="#{jobParameters['useruf']}" /> <beans:property name="gridsize” value="#{jobParameters[' gridsize ']}" /> <beans:property name="node” value="#{jobParameters['node']}" />

<beans:property name="date" value="#{jobParameters['date']}" /> <beans:property name="lastDateTrait" value="#{jobParameters['lastDateTrait']}" /> </beans:bean>

Hope that help some one it is just on one machine think if we have multiple machines like Linux Beowulf or a microsoft network hope that the database support the load.

Ok to run multiple jobs at the same time please visit

http://forum.spring.io/forum/spring-projects/batch/53803-executing-multiple-jobs-at-the-same-time

Finally Excuse my English it is not my mother tongue and execuse the doc is not well presented I do it in a hurry I didn’t pay attention to the layout

--

Minds, like parachutes, function best when open. ,,,

(o o)

/ -------------------oOO--(_)--OOo---------------------\r

| Wadï Mami didipostman

| e-mail : wadi.mami@gmail.com

| -----------------------------------------------------/

|

| |

--------------------------/ ------------------------/